Distributed control of video encoding speeds

ABSTRACT

Video encoding operation is performed in a distributed video encoding system to meet a service layer agreement such as a target time for completion of the video encoding operation. Each of multiple videos being encoded is split into smaller jobs corresponding to video segments of the videos. The jobs are processed according to a job queue. The sequence of jobs in the job queue is periodically updated such that jobs corresponding to each video are processed within the corresponding target encoding time.

BACKGROUND

Universal availability of digital video cameras and storage products hasenabled new uses of digital video. In contrast to the traditionalnetwork delivery of video, e.g., digital satellite or cable television,more and more video is being captured by users using their cameras andsmartphones and shared over the Internet with others.

While the availability of internet-connected video capturing devices hassoared, the ability of these devices to encode, or compress, and uploadthe captured audio/video content remains limited. Further, thesedevices, such as internet-connected cameras, tablets and smartphones,may frequently be offline and thus rely on a service provider to makecontent captured from the devices available to others at the time andplace others want to view the content.

BRIEF DESCRIPTION OF THE DRAWINGS

The techniques introduced here may be better understood by referring tothe following Detailed Description in conjunction with the accompanyingdrawings, in which like reference numerals indicate identical orfunctionally similar elements:

FIG. 1 illustrates a prior art video encoding system;

FIG. 2 illustrates an example of a distributed video encoding system;

FIG. 3 illustrates another example of a distributed video encodingsystem;

FIG. 4 illustrates an example method of placing key frames at desiredlocations in an output video bitstream;

FIG. 5 illustrates an example of a distributed video encoding system forcomputing and allocating bits on a per frame basis using a complexityanalyzer;

FIG. 6 illustrates another example of a distributed video encodingsystem for computing and allocating bits on a per frame basis using acomplexity analyzer;

FIG. 7 another example of a distributed video encoding system forcomputing and allocating bits on a per frame basis using a complexityanalyzer;

FIG. 8 illustrates an example of a method for computing and allocatingbits on a per frame basis in a distributed video encoding using acomplexity analyzer;

FIG. 9 illustrates an example of a method for computing and allocatingbits per frame in a distributed video encoding using a separatecomplexity analyzer for each segment of a video;

FIG. 10 illustrates an example of a method for computing and allocatingbits per frame in a distributed video encoding by using a separatecomplexity analyzer for each segment of a video and adding a pathwaythat passes each segment complexity measurement to a correspondingencoder;

FIG. 11 illustrates an example of a communication network for videouploading and sharing;

FIG. 12 illustrates example graphs illustrating relationships betweenencoding time and quality and video bitrate and encoding effort;

FIG. 13 illustrates an example time line of video uploading;

FIG. 14 illustrates an example method for uploading digital video from auser device to a network server;

FIG. 15 illustrates an example of a distributed video encoding workflow;

FIG. 16 illustrates an example of a video frame encoding sequence;

FIG. 17 illustrates another example of a video frame encoding sequence;

FIG. 18 illustrates an example of operation of an intelligent segmenter;

FIG. 19 illustrates a method for encoding a video by changing positionsof at least some key frames in the resulting encoded bitstream;

FIG. 20 illustrates an example graph of relationship between encoderparameter settings and quality of encoded video;

FIG. 21 illustrates an example system for performing ultra-high videoencoding using multiple encoders;

FIG. 22 illustrates an example flowchart of a method for encoding avideo;

FIG. 23 illustrates an example of destabilization caused in video framesdue to camera movement;

FIG. 24 illustrates an example block diagram for generating a transformfile from an encoded video;

FIG. 25 illustrates an example block diagram of distributed imagestabilization of a video;

FIG. 26 illustrates an example block diagram of performing distributedimage stabilization using multiple worker nodes;

FIG. 27 illustrates an example block diagram of receiving results ofimage stabilization from multiple worker nodes;

FIG. 28 illustrates an example block diagram of generating an imagestabilized video at a master node;

FIG. 29 illustrates an example flowchart of a method of distributedimage stabilization;

FIG. 30 illustrates an example of a distributed video encoding system;

FIG. 31 illustrates an example flowchart of a decision process for nodeselection performed at a master node in a distributed video encodingsystem;

FIG. 32 illustrates an example flowchart of a decision process at aworker node in a distributed video encoding system;

FIG. 33 illustrates another example flowchart of a decision process at aworker node in a distributed video encoding system;

FIG. 34 illustrates an example flowchart of a method of distributedvideo encoding by assigning video segments to worker nodes;

FIG. 35 illustrates an example of a video encoding pipeline in adistributed video encoding system;

FIG. 36 illustrates an example flowchart of a method of distributedencoding of multiple videos;

FIG. 37 illustrates an example server apparatus;

FIG. 38 illustrates an example of video complexity as a function oftime.

While the flow and sequence diagrams presented herein show anorganization designed to make them more comprehensible by a humanreader, those skilled in the art will appreciate that actual datastructures used to store this information may differ from what is shown,in that they, for example, may be organized in a different manner; maycontain more or less information than shown; may be compressed and/orencrypted; etc.

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of the claimed embodiments.Further, the drawings have not necessarily been drawn to scale. Forexample, the dimensions of some of the elements in the figures may beexpanded or reduced to help improve the understanding of theembodiments. Similarly, some components and/or operations may beseparated into different blocks or combined into a single block for thepurposes of discussion of some of the embodiments. Moreover, while thevarious embodiments are amenable to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and are described in detail below. Theintention, however, is not to limit the particular embodimentsdescribed. On the contrary, the embodiments are intended to cover allmodifications, equivalents, and alternatives falling within the scope ofthe disclosed embodiments as defined by the appended claims.

DETAILED DESCRIPTION

Various of the disclosed embodiments disclose allocating a number ofbits on a per-frame basis in a distributed video encoding using acomplexity analyzer. The analyzer receives an input video for allocatingbits for each segment of the input video based on the complexity of thesegment and splits the input video into multiple segments. The analyzerdetermines the bits per frame to be allocated to the multiple segmentsbased on the complexity measurement of the input video. In someembodiments, a single complexity analyzer can be used to determine thecomplexity of the plurality of segments. In another aspect, a separatecomplexity analyzer can be used for each segment to determine thecomplexity. The analyzer can allocate the number of bits to the multiplesegments of the input video and combine the multiple segments to form asingle output video; thereby, obtaining the encoded output video.

Various of the disclosed embodiments disclose encoding a segment of aninput video with accurate placement of key frames, e.g., I-Frames, usingGroup of Pictures (GOPs) length required for output in a distributedvideo encoding system with two or more encoders. The encoder createsoverlapping segments to allow precise key frame placement, uses thedesired segment length and the desired key frame placement to create theoverlapping segments, and can skip a number of frames at the beginningof the segment and a certain number of frames at the end of the segment,thereby achieving accurate placement of key frames.

Various of the disclosed embodiments disclose techniques in which videois transmitted from a client device to a server in the network byminimizing the amount of time it takes to upload the video from theclient device to the server while being encoded from a first videoencoding format in which the video is stored at the client devices to asecond video encoding format specified by a user and/or by the server.Operational conditions such as fullness of a transmission buffer at theclient device and the target quality of encoded video are used tocontrol the video encoding operation.

Various of the disclosed embodiments relate to a distributed videoencoding system that splits an input video into video segments. Thevideo segments are encoded using multiple video encoding nodes. Prior tosplitting the video into video segments, the video is analyzed to ensurethat each video segment includes all the video frames from which othervideo frames within that segment have been encoded in the input video.For example, picture headers are inspected to determine the temporaldistance of the farthest past and future reference frames used forencoding frames of a video segment.

Various of the disclosed embodiments describe techniques in whichmultiple video encoders are used to simultaneously encode a video usingencoders configured using different encoding parameters. A segmentselector selects an encoded version of the encoded video segment usingoperational criteria such as video quality and bandwidth. Aconfiguration determination module may analyze the video segment to makea decision about which encoding parameter configurations may be suitablefor encoding the video segment. The configuration determination modulemay be trainable, based on past encoding results.

Various of the disclosed embodiments perform stabilization of images ina video. In a distributed video encoding system, a video is encoded bysplitting into video segments and encoding the segments using multipleencoders. Prior to segmenting the video for distributed video encoding,image stabilization is performed on the video. For each frame in thevideo, a corresponding transform operation is first computed based on anestimated camera movement. Next, the video is segmented into multiplevideo segments and the corresponding per-frame transform information forthe multiple video segments. The video segments are then distributed tomultiple processing nodes that perform the image stabilization of thecorresponding video segment by applying the corresponding transform. Theresults from all the stabilized video segments are then stitched backtogether for further a video encoding operation.

Various of the disclosed embodiments relate to a distributed videoencoding or transcoding system that may utilize multiple encoding nodesto encode/transcode a video by splitting the video into multiple smallervideo segments. The assignment of video segments to the encoding nodesis performed to balance the use of the encoding nodes by selecting anode based on its encoding capabilities, e.g., whether the node employeda central processing unit (CPU) based encoding or graphics processorunit (GPU) based encoding.

Various of the disclosed embodiments disclose video encoding ortranscoding operation in a distributed video encoding system to meet aservice layer agreement such as a target time for completion of thevideo encoding operation. Each of multiple videos being encoded is splitinto smaller jobs corresponding to video segments of the videos. Thejobs are processed according to a job queue. The sequence of jobs in thejob queue is periodically updated such that jobs corresponding to eachvideo are processed within the corresponding target encoding time.

In various embodiments, distributed video and image processing may beperformed using computational resources that are available on hardwareplatforms that are communicatively coupled to each other via networkconnections. The network between two such computational hardwareplatforms may be geographically distributed, and the communication maytravel over networks that may or may not be controlled by the sameentity that controls the availability and use of the computationalresources. For example, the computational resources may include serversavailable in the cloud at unspecified or unknown locations, e.g., atlocations not known to the video encoders. As another example, thecomputational resources may be available on servers available at datacenters managed by a social networking service provider.

Video is being produced and consumed more than ever, and in moreformats, device types and variety of networks than ever. Transcoding, orencoding, is a process of translating or converting a video encoded fromone format to another. In some embodiments, only bitrate of the videobitstream may be changed. Encoding is a computationally intensiveprocess. Due to the proliferation of video content used, distributedcomputing approach has been applied to video encoding to exploit theextra computational resources available among multiple machines,multi-core CPUs, and distributed computing resources in a givenfacility, home or a dedicated cloud computing infrastructure, and so on.This distributed encoding approach has been helpful in reducingrendering and start-up time of a video for on demand encoding.

Encoding systems take an input video, composed of many frames (I, P, orB frame types), and create a new output video, often with a differentoutput resolution or bit-rate and often with different frame types. Innormal encoding systems, there is no relation between the placement ofI-Frames in the input video and the output video; the encoding systemcan output frames in any order and type specified by the user. This isimportant because the type of frames used has a significant impact onthe output bit-rate and quality, and selection is done based on outputbit-rate and/or quality requirements.

Traditional video encoding systems use a single encoder to encode inputvideo. In this implementation, it is easy to configure the system toplace I-Frames accurately in the output video. As the encoder is workingthrough the input video, it can make a decision for the frame type foreach frame of the output video based on user parameters.

Distributed Encoding Systems enable multiple different computers to worktogether to create a single output video by encoding individual segmentsof the video before the encoded segments are joined together into asingle output vide. The input video is broken into output segments,where the segments are divided at I-Frames. When a computer takes aninput video segment and creates an output video segment, it must startthe output video segment with an I-Frame. This means that there is now arelation between the placement of the I-Frames in the input video and inthe output video, and it makes it difficult to guarantee accurateI-Frame placement in the output video, using known mechanisms.

Various examples of the disclosed techniques will now be described infurther detail. The following description provides specific details fora thorough understanding and enabling description of these examples. Oneskilled in the relevant art will understand, however, that thetechniques discussed herein may be practiced without many of thesedetails. Likewise, one skilled in the relevant art will also understandthat the techniques can include many other obvious features notdescribed in detail herein. Additionally, some well-known structures orfunctions may not be shown or described in detail below, so as to avoidunnecessarily obscuring the relevant description.

The terminology used below is to be interpreted in its broadestreasonable manner, even though it is being used in conjunction with adetailed description of certain specific examples of the embodiments.Indeed, certain terms may even be emphasized below; however, anyterminology intended to be interpreted in any restricted manner will beovertly and specifically defined as such in this section.

Overview—Example Network Topology

Turning now to the figures, FIG. 11 Illustrates an example of acommunication network 1100 in which a distributed network 1102 isoperating to facilitate the use of digital video among multipleusers—user 1104 who may be the originator of a video using his userdevice 1106 and users 1114 with their user devices 1116, who maydownload and view the video sequence that the user device 1106 uploadsto a server 1110 in the distributed network 1102. The user devices 1106,1116 may communicate with the distributed network 1102 via communicationnetworks or channels 1108, 1112. The channels 1108, 1112 may be same ordifferent networks, e.g., the Internet or World Wide Web, and may changefrom time to time based on location. For example, user devices 1106,1116 may include multiple network modules, e.g., a Wi-Fi modem, a 3Gmodem, a 4G modem, a WiMax modem, etc. and may use one or more of thenetwork connections for communication in the upstream direction (fromthe device to the network) or the downstream direction (from the networkto the device).

In one example use case, the user 1104 may capture a video clip using acamera-enabled smartphone (user device 1106). The user 1104 may theninstruct the user device 1106 to upload the video to the server 1110,which may be a server operated by a service provider, e.g., a socialmedia website. Once the user device 1106 uploads the video to the server1110, the service provider may operate the distributed network 1102,e.g., a geographically distributed server farm, to propagate theavailability of the user's video clip to other users with whom the user1104 wishes to share the video clip, e.g., user 1114.

In conventional video upload and sharing systems, a user typicallycannot control the video upload process. For example, many users capturevideos using their mobile phones, but cannot control the quality orother encoding parameters when the captured video is converted into acompressed digital video bitstream. Further, the user may be able toinitiate video uploading to a social media website or to a serviceprovider's file transfer protocol (ftp) site, but may not be able tocontrol operational characteristics of the upload such as the uploadspeed or quality throttling of the uploading process in real time.

Furthermore, depending on the user's operating condition, the user'spreferences for how the video should be uploaded may change. Forexample, when the user is on the go, the user may want uploading tohappen as fast as possible, where video quality may be sacrificed tomeet faster upload performance. As another example, in anotheroperational scenario, the user may want to upload the video by encodingat the highest possible quality, e.g., when the user device is operatingin a Wi-Fi broadband network. In yet another operating scenario, theuser may incur byte-based uploading costs and thus may want to upload avideo using the minimum file size while still meeting some low qualitythreshold. In some embodiments, encoded video quality may be heldconstant, but encoding effort may be varied based on the availablebandwidth for upload.

Many service providers today operate distributed computer networks thatcan propagate user-uploaded files to geographically distributed filestores for access from anywhere in the world. These service providersmay also interface with and rely on content delivery network (CDN)infrastructure such as CDNs operated by Akamai or LimeLight Networks.When it comes to the distribution of user-uploaded videos though, theservice providers often fail to fully utilize the availablecomputational power in a distributed network.

In traditional distributed video encoding systems as shown in FIG. 1,the input video is split into multiple segments by a splitter module.Each video segment is then passed onto an encoder, where the encoderprocesses the received video segment into an output video segment (i.e.,changing the video segment from the source format to other format whichis compatible with any other device). The output video segments from allthe video encoders are combined together as shown in the FIG. 1;thereby, achieving a single output video.

For example, if the input video has the I-Frame for every 60 frames thenthere will be an I-Frame at frame 0, 60, 120, 180, 240, 300, 360, etc.The distributed video encoding system will break up the input video atpredetermined points, and pass each segment to the separate encoder. Ifthe configured split size is 600 frames, each encoder will receive 600frames to encode. If the user wants the output video to have an I-Frameevery 250 frames, it is not possible for the system to follow thisrestriction. Each encoder will place the I-Frame at frame 0, frame 250,and frame 500. When 2 of these pieces are combined together, the outputvideo will have frames at 0, 250, 500, 600, 850, and 1100; thus notachieving the task of placing an I-Frame every 250 frames as specifiedby the user

Thus, it would be beneficial to accurately place I-Frames in the outputvideo when a distributed video encoding system is used to produce theoutput video.

The techniques disclosed herein can be used to, among other uses,address the shortcomings of present-day video uploading and processingin a distributed network. These, and other, detail are disclosed in thepresent document.

In some embodiments, input video is segmented into overlapping segmentsthat each contain the requisite data to construct an output segment thatbegins with a key frame (e.g., an independently decodable frame) and hasa desired length. The overlapping segmentation is performed such thateverywhere there is a desired output key frame, there will be enoughdata after that location to have a complete segment and enough databefore that location to have an input key frame. To achieve this, everyencoder is configured to ignore extra video frames before and/or afterthe desired output segments.

FIG. 2 illustrates a high level overview of a distributed video encodingsystem 200 to accurately place I-Frames in an output video using acontroller module, according to the embodiments as disclosed herein. Inan embodiment, the distributed video encoding system 200 includes asplitter module 102, a controller module 202, one or more encodermodules 104 _(1-N) (hereafter referred as encoder module(s) 104), and acombiner module 106.

The splitter module 102 can be configured to receive an input videoincluding an I-Frame for some number of (e.g., 60) frames of the inputvideo that is to be encoded from a source format to any other format.The splitter module 102 splits the received input video into a pluralityof segments each including a specified number of frames. Here, thesplitter module 102 can be configured to receive information from thecontroller module 202 to split the received input video into pieces oflength (i.e., segment length) having 600 frames and also placement ofI-Frames for every 250 frames (Output Group of Pictures (GOP) length) ofthe input video. For example, the splitter module 102 splits the inputvideo into a video segment-1, a video segment-2, and a video segment-3as shown in the FIG. 2. Here, the video segment-1 includes 0 to 539frames of the input video. The video segment-2 includes 480 to 1019frames of the input video. The video segment-3 includes 960 to 1500frames of the input video. Each segment includes one or more overlappingframes at the beginning of the segment with immediately precedingsegment, and one or more overlapping frames at the end of the segmentwith immediately following segment. Also, the two sequential segmentscarry the starting frame of the overlapping GOP.

The splitter module 102 determines a number of output GOPs that can fitinto each segment length. Here, the splitter module 102 determines 2output GOPs for each segment of the input video. In an embodiment, thenumber of frames in each segment can be based on for example, a scenechanges in the input video. For example, if the video segment containsthe scene change, the frames before the scene change could bedrastically different than the frames after the scene change. Theencoding of the segment may involve significant challenges. In thiscase, it would be desirable to alter the length of the segment such thatit may not include the scene change for the effectively performing theencoding.

The splitter module 102 segments entire input GOPs to the segment untilthe segment has enough frames to create the desired number of outputGOPs. The splitter module 102, for the first segment, 540 frames (i.e.,start frame 0 to 539 frames) can be segmented so that the 2 output GOPsof length 250 each (i.e., 500 frames total) can be supported. Thesplitter module 102, for the second segment and all subsequent segments,the algorithm will repeat this process, but it will first rewind by oneor more GOPs in the input. The second segment starts at frame 480 sothat the segment includes frame 500, which is used for the 2nd outputGOP. The third segment starts at frame 960 so that the segment includesframe 1000, which is used for the 3rd output GOP. The splitter module102 sends the length of each segment, and the start and stop frame ofeach segment to the controller module 202.

Unlike conventional systems, the controller module 202 can be configuredto send information to split the received input video into variouspieces of length (e.g., segment length) and also placement of I-Framesin the input video. The controller module 202 can be configured to senda number of frames to skip at the beginning of each video segment, whereto place each I-Frame, and a number of frames to skip at the end of eachvideo segment to the encoder module 104.

Further, the splitter module 102 outputs the video segments (i.e., videosegment-1, video segment-2, and video segment-3) to the encoder modules104. After receiving multiple segments from the splitter module 102, theencoder module 104 can encode according to a Windows Media Video or VC-1format, MPEG-x format (e.g., MPEG-1, MPEG-2, or MPEG-4), H.26x format(e.g., H.261, H.262, H.263, or H.264), or other format. In anembodiment, the encoder module 104 receives the number of frames to skipat the beginning of the frame and at the end of the frame from thecontroller module 202 as shown below:

Video Segment-1 (0 to 539 frames): Skip “0” frame at the beginning ofthe video segment and place the I-Frame at every 250 frames of the videosegment. Skip “40” frames at the end of the video segment.

Video Segment-2 (480 to 1019 frames): Skip “20” frames at the beginningof the video segment and place the I-Frame at every 250 frames. Skip“20” frames at the end of the segment.

Video Segment-3 (960 to 1500 frames): Skip “40” frames at the beginningof the video segment and place the I-Frame at every 250 frames. Skip “0”frames at the end of the segment.

Further, the encoder module 104 then processes each video segmentreceived from the splitter module 102 and by using the information fromthe controller module 202 to determine how many frames to skip and thenskips the frames at the beginning of the frame and at the end of theframe for accurate placement of I-Frames. The video segments from theencoder module 104 are sent to the combiner module 106. On receiving thevideo segments from the encoder module 104, the combiner module 106combines the plurality of segments to form the single encoded outputvideo with accurate placement of I-Frames.

FIG. 2 illustrates a limited overview of a distributed video encodingsystem 200 but it is to be understood that another embodiment is notlimited thereto. Further, the system 200 can include any number of videoprocessing modules and techniques along with other hardware or softwarecomponents communicating with each other. For example, the component canbe, but not limited to, a process running in the controller orprocessor, an object, an executable process, a thread of execution, aprogram, or a computer. By way of illustration, both an applicationrunning on a device and the device itself can be a component.

FIG. 3 illustrates another high level overview of a distributed videoencoding system 300 to accurately place I-Frames in an output video,according to embodiments as disclosed herein. In an embodiment, thedistributed video encoding 300 includes a splitter module 102, acontroller module 202, one or more encoder modules 104 _(1-N) (hereafterreferred as encoder module(s) 104), and a combiner module 106.

The detailed operations of each module included in the FIG. 3 areexplained in conjunction with the FIG. 2. The FIG. 3 illustrates alimited overview of a distributed video encoding system 300 but it is tobe understood that another embodiment is not limited thereto. Further,the system 300 can include any number of video processing modules andtechniques along with other hardware or software componentscommunicating with each other. For example, the component can be, butnot limited to, a process running in the controller or processor, anobject, an executable process, a thread of execution, a program, or acomputer. By way of illustration, both an application running on adevice and the device itself can be a component.

FIG. 4 is a flow diagram illustrating a method to accurately placeI-Frames in an output video using a controller module in a distributedvideo encoding system, according to embodiments as disclosed herein. Atblock 402, the method 400 includes receiving an input video forconverting from a source format to any other format with accurateplacement of I-Frames. The method 400 allows the splitter module 102 toreceive the input video for converting from a source format to any otherformat with accurate placement of I-Frames. At block 404, the method 400includes receiving the information to split the input video into apredetermined segment length and also placement of I-frames from thecontroller module 202. Unlike conventional systems, the method 400allows the splitter module to receive the information to split the inputvideo into a predetermined segment length and also placement of I-framesfrom the controller module 202. For example, the splitter module 102receives information from the controller module 202 to split thereceived input video into pieces of length (i.e., segment length) 600frames and also placement of I-Frames for every 250 frames (Output Groupof Pictures (GOP) length) of the input video.

At block 406, the method 400 includes determining the segment lengthbased on a number of GOPs that can be included in each segment. Themethod 400 allows the splitter module 102 to determine the segmentlength based on the number of GOPs that can be included in each segmentof the input video. At block 408, the method 400 includes segmenting thereceived input video into plurality of segments after determining thesegment length. The method 400 allows the splitter module 102 todetermine the segment length based on the number of GOPs that can beincluded in each segment of the input video. In an embodiment, thesplitter module 102 splits the input video into a video segment-1, avideo segment-2, and a video segment-3. Here, the video segment-1includes 0 to 539 frames of the input video. The video segment-2includes 480 to 1019 frames of the input video. The video segment-3includes 960 to 1500 frames of the input video. The splitter module 102determines 2 output GOPs that can fit into each segment length.

At block 410, the method 400 includes sending the input video along withthe length of the video segment, start and stop frames for each segmentto be encoded. Unlike conventional systems, the method 400 allows thesplitter module 102 sends the input video along with the length of thevideo segment, start and stop frames for each segment to be encoded tothe controller module 202 to decide on the number of frames to skip atthe beginning and at the end of each segment. Also, the splitter module102 sends the plurality of segments to the encoder modules 104. At block412, the method 400 includes sending the information to plurality ofencoders on the I-Frame position, number of frames to skip at thebeginning of the video segment, and number of frames to skip at the endof the video segment. The method 400 allows the encoder module 104 toreceive the information to plurality of encoders on the I-Frameposition, number of frames to skip at the beginning of the videosegment, and number of frames to skip at the end of the video segment toaccurate place I-Frames in the video segment.

In an embodiment, a separate encoder can be used to receive each segmentof the video for transferring the segment from one format to anotherformat while achieving accurate placement of the I-Frames in the inputvideo. For example, consider an encoder-1, an encoder-2, and anencoder-3 receives an input video split into three segments namely,segment-1, segment-2, and segment-3 respectively. Also, the encoder-1receives information to skip ‘0” frames at the beginning of the videosegment and placing the I-Frame at every 250 frames of the videosegment. Skip “40” frames at the end of the video segment. The encoder-2receives information to skip “20” frames at the beginning of the videosegment and placing the I-Frame at every 250 frames. Skip “20” frames atthe end of the segment. The encoder-3 receives Skip “40” frames at thebeginning of the video segment and place the I-Frame at every 250frames. Skip “0” frames at the end of the segment. The encoder-1 Unlikeconventional systems, different encoder's uses information received fromthe controller module 202 to accurately place I-Frames in each segmentof the video such as to generate a uniform quality of video.

Further, at block 414, the method 400 includes encoding the receivedsegments and places the I-Frames using the information from thecontroller module 202. The method allows the encoder module 104 toencode the received segments and places the I-Frames using theinformation from the controller module 202. At block 416, the method 400includes combining the plurality of segments received from the encodermodule 104 to form a single encoded output video with accurate placementof I-Frames. The method 400 allows the combiner module 106 to combinethe plurality of segments to form the single output video. For example,the segment-1 from the encoder-1, the segment-2 from the encoder-2, andthe segment-3 from the encoder-3 are combined to form the single outputvideo. Unlike conventional systems, creating overlapping segments inorder to allow precise I-Frame placement in the input video. Thesplitter uses the desired segment length and the desired I-Frameplacement to create overlapping segments. The encoder skips the numberof frames at the beginning and at the end of the video segment foraccurate placement of I-Frames.

The various actions, acts, blocks, steps, and the like in the method 400may be performed in the order presented, in a different order orsimultaneously. Further, in some embodiments, some actions, acts,blocks, steps, and the like may be omitted, added, modified, skipped,and the like without departing from the scope of the invention.

Traditionally, a variable bit-rate (VBR) bitstream is created using asingle-pass encoding or a multi-pass encoding. The single-pass encodinganalyzes and encodes the data “on the fly” and it is used in a constantbit-rate (CBR) encoding. The single-pass encoding is used when theencoding speed is most important (e.g., for real-time encoding). Thesingle-pass VBR encoding is usually controlled by the fixed qualitysetting or by the bit-rate range (i.e., minimum and maximum allowedbit-rate) or by the average bit-rate setting. The multi-pass encoding isused when the encoding quality is most important. The multi-passencoding may not be used in real-time encoding and live broadcast orlive streaming scenarios.

The multi-pass encoding is typically used for the VBR encoding becausethe CBR encoding doesn't offer any flexibility to change the bit-rate.VBR encoding allows a higher bit-rate to be allocated to the morecomplex segments of the input video while a lower bit-rate is allocatedto the less complex segments of the input video. The average of thesebit-rates can be calculated to produce the average bit-rate for thevideo. The most common VBR encoding is a two-pass encoding. In the firstpass of the two-pass encoding, the input data is being analyzed tolocate the scenes that have higher complexity (i.e., which require alarger amount of bits in order to achieve the desired quality) and theresult is stored in a log file. In the second pass, the collected datafrom the first pass is used to allocate bits-per-frame to each frame inthe video for achieving the best video encoding quality. In the videoencoding process, the two-pass encoding is usually controlled by theaverage bit-rate setting or by the bit-rate range setting (e.g., minimaland maximal allowed bit-rate) or by the target video file size setting.

However, in the distributed video encoding process, the input video canbe split into multiple segments and the multiple segments are assignedto different computers to perform the video encoding process. Here, eachcomputer, on receiving a video segment, performs video encoding withinthe single segment available to it. The distributed video encodingsystem performing video encoding using variable number of bits per framewithin the single segment may not achieve good encoding performanceacross multiple segments as each computer is working on a single segmentand thus cannot access information about complexity in other segments tobetter allocate a number of bits per frame. For example, consider ascenario where the input video has a high complexity region in themiddle of the video. The high complexity region of the video isallocated (in a non-distributed system) with additional bits to delivergood video quality. In a traditional distributed video encoding systemwhere the input video is split into multiple segments, each segmentencoder achieves the average bit-rate for its segment and therefore atthe video level, bits are undesirably allocated to the high and lowcomplexity segments of the video equally.

In traditional distributed video encoder systems as shown in FIG. 1, theinput video is split into multiple segments by a splitter module. Eachvideo segment is then passed onto an encoder, where the encoderprocesses the received video segment into an output video segment (i.e.,changing the video segment from the source format to other format whichis compatible with any other device). The output video segments from allthe video encoders are combined together as shown in the FIG. 1;thereby, achieving a single output video.

Thus, it is beneficial to accomplish high quality encoding by allocatinga variable number of bits per frame in the distributed video encodingsystem; thereby, maintaining uniform quality across the video.

FIG. 5 illustrates a high level overview of a distributed video encoding500 for allocating a variable number of bits per frame in a distributedvideo encoding using a complexity analyzer, according to the embodimentsas disclosed herein. In an embodiment, the distributed video encoding500 includes a splitter module 502, a complexity analyzer module 504,one or more encoder modules 506 _(1-N) (hereafter referred as encodermodule(s) 506), and a combiner module 508.

The splitter module 502 can be configured to receive an input video thatis to be encoded from a source format to any other format. Here, theinput video generally refers to a stream including both audio componentsand video components. The splitter module 502 splits the received inputvideo into a plurality of segments each including a certain number offrames. For example, the splitter module 502 splits the input video intoa video segment-1, a video segment-2, and a video segment-3. In anembodiment, each segment includes 60 frames. In another embodiment, eachsegment can vary across a range of values such as including between 30frames to 90 frames. The number of frames in each segment can be basedon for example, a scene changes in the input video. For example, if thevideo segment contains the scene change, the frames before the scenechange could be drastically different than the frames after the scenechange. The encoding of the segment may involve significant challenges.In this case, it would be desirable to alter the length of the segmentsuch that it may not include the scene change for the effectivelyperforming the encoding.

Further, the splitter module 502 can be configured to output the videosegments to the encoder modules 506. After receiving multiple segmentsfrom the splitter module 502, the encoder module 506 can encodeaccording to a Windows Media Video or VC-1 format, MPEG-x format (e.g.,MPEG-1, MPEG-2, or MPEG-4), H.26x format (e.g., H.261, H.262, H.263, orH.264), or other format. The encoder module 206 can encode according toone or more audio standards such as WAV, FLAC, MP3, WMA, or some otherstandard. The encoder module 506 can also receive the number of bits tobe allocated to each segment of the input video from the complexityanalyzer module 504.

The complexity analyzer module 504 can be configured to receive theinput video that is to be encoded from a source format to any otherformat. The complexity analyzer module 504 can be configured to processthe entire video to measure the complexity of the video at every frameof the video. After determining the complexity at every frame of thevideo, the complexity analyzer module 504 can be configured to computethe bits-per-frame to be allocated to each segment. Thus, a givensegment can be encoded at a bitrate independently from any othersegment. Unlike conventional systems, the complexity analyzer module 504can be configured to send the bits per frame allocated to each segmentto the respective encoder module 506, in order to achieve a uniformvideo quality across segments of the video.

Further, the encoder module 506 receives the video segments from thesplitter module 502 and also receives the bitrate settings to be used ineach segment for generating a uniform video quality. Each segment isallocated with the bitrate settings received from the splitter module502 based on the complexity associated thereof. The combiner module 508receives the output video segments from the encoder modules 506 andcombines the output video segments to generate the encoded video output.

FIG. 5 illustrates a limited overview of a distributed video encodingsystem 500 but it is to be understood that another embodiment is notlimited thereto. Further, the system 500 can include any number of videoprocessing modules and techniques along with other hardware or softwarecomponents communicating with each other. For example, the component canbe, but not limited to, a process running in the controller orprocessor, an object, an executable process, a thread of execution, aprogram, or a computer. By way of illustration, both an applicationrunning on a device and the device itself can be a component.

FIG. 6 illustrates another high level overview of a distributed videoencoding system 600 for computing and allocating a variable number ofbits per frame using a separate complexity analyzer for each segment ofa video, according to embodiments as disclosed herein. In an embodiment,the input video is split into plurality of segments including a set offrames. Each segment of the video is passed through a separatecomplexity analyzer module 604, which produces an output file thatdescribes the video complexity for each frame of the video. Once thecomplexity analyzer modules 604 determine the complexity of eachsegment, a bit-rate for each video segment is computed and is passed tothe encoder module 506 to achieve a desired quality of video. Further,detailed operations by the system 600 are described in conjunction withthe FIG. 9.

FIG. 6 illustrates a limited overview of the system 600 but it is to beunderstood that another embodiment is not limited thereto. Further, thesystem 600 can include any number of video processing modules andtechniques along with other hardware or software componentscommunicating with each other. For example, the component can be, butnot limited to, a process running in the controller or processor, anobject, an executable process, a thread of execution, a program, or acomputer. By way of illustration, both an application running on adevice and the device itself can be a component.

FIG. 7 illustrates another high level overview of a distributed videoencoding system 400 for computing and allocating a variable number ofbits per frame using a separate complexity analyzer for each segment ofa video and adding a pathway that passes each segment complexitymeasurement to a corresponding encoder, according to embodiments asdisclosed herein. In an embodiment, the encoder module 206 can use thesegment complexity information computed earlier to make effective localdecisions within the segment. Further, the various operations performedby the distributed video encoding system 700 are described inconjunction with the FIG. 10.

FIG. 7 illustrates a limited overview of the system 700 but it is to beunderstood that another embodiment is not limited thereto. Further, thesystem 700 can include any number of video processing modules andtechniques along with other hardware or software componentscommunicating with each other. For example, the component can be, butnot limited to, a process running in the controller or processor, anobject, an executable process, a thread of execution, a program, or acomputer. By way of illustration, both an application running on adevice and the device itself can be a component.

FIG. 8 is a flow diagram illustrating a method 800 for computing andallocating a variable number of bits per frame (or simply “bits”) in adistributed video encoding using a complexity analyzer, according toembodiments as disclosed herein. At block 802, the method 800 includesreceiving an input video for allocating bits for each segment based onthe complexity of the segment. At block 804, the method 800 includessplitting the input video into a plurality of segments. The method 800allows the splitter module 602 to split the input video into theplurality of segments including a set of frames.

At block 806, the method 800 includes determining the bits to beallocated to the plurality of segments based on the complexity of theinput video. The method 800 allows the complexity analyzer module 804 todetermine the bits to be allocated to the plurality of segments based onthe complexity measurement of each segment of the input video. Here, thecomplexity analyzer module 804 measures the complexity of every frame ineach segment and also complexity of the entire input video to measureoverall complexity of the segment to determine the bits to be allocatedto each segment of the video. For example, consider an input video issplit into 3 segments i.e., Segment-1, Segment-2, and Segment-3. Unlikeconventional systems, a complexity associated with each segment (theSegment-1, Segment-2, and Segment-3) is analyzed and accordingly a bitbudget for each segment is computed to generate a uniform video quality.

At block 808, the method 800 includes allocating the bits to pluralityof segments of the input video for encoding the plurality of segmentsinto a different format. Based on the complexity of each segment, themethod 800 allows the encoder module 506 to allocate the bits to theplurality of segments. Here, the encoder module 506 receives theplurality of segments of the input video from the splitter module 202and also receives the bits to be allocated to the plurality of segments.The encoder module 506 accepts the information from the complexityanalyzer module 604 and accordingly allocates the bits to each segmentto encode the each segment into a different format. For each segment theencoder will achieve an average bit rate for the segment, and canallocate bits to the high and low complexity sections of the video. Forexample, a high complexity region in a segment is allocated withadditional bits in order to deliver good video quality. Similarly, a lowcomplexity region in a segment is allocated with uniform bits so as todeliver the good video quality.

In an embodiment, a separate encoder can be used to receive each segmentof the video for transferring the segment from one format to anotherformat while achieving a desired video quality. For example, consider anencoder-1, an encoder-2, and an encoder-3 receives an input video splitinto three segments namely, segment-1, segment-2, and segment-3respectively. The encoder-1 allocates the bits to the Segment-1 based onthe complexity of the segment-1. Similarly, the encoder-2 allocates thebits to the Segment-1 based on the complexity of the segment-2.Similarly, the encoder-3 allocates the bits to the Segment-1 based onthe complexity of the segment-3. Unlike conventional systems, differentencoder's uses variable bit rates encoding each segment of the videosuch as to generate a uniform quality of video. Each bit rate for thesegment is computed based on the complexity analysis of the segment.

Further, at block 810, the method 800 includes combining the pluralityof segments to form a single output video. The method 800 allows thecombiner module 208 to combine the plurality of segments to form thesingle output video. For example, the segment-1 from the encoder-1, thesegment-2 from the encoder-2, and the segment-3 from the encoder-3 arecombined to form the single output video. The combining may be performedby concatenating the segments in the same temporal order as in thereceived video.

At block 812, the method 800 includes obtaining a encoded output videofrom the combiner module 508. Unlike conventional systems, distributedvideo encoding system obtains the encoded output video while maintaininga desired quality by using variable bit rates for each segment insteadof a globally optimized bit rate. The complexity of the input video inthe distributed video encoding is measured and the bit budget on aper-frame basis is determined for each segment of the input video. Theinformation about the per frame bit allocation is passed onto theencoder for allocating bits to each segment thereby obtaining goodaverage video quality.

FIG. 9 is a flow diagram illustrating a method 900 for allocatingencoding bits in a distributed video encoding system using a separatecomplexity analyzer for each segment of a video, according to theembodiments as disclosed herein. At block 902, the method 900 includesreceiving an input video for allocating bits for each segment based onthe complexity of the segment. At block 904, the method 900 includessplitting the input video into a plurality of segments. The method 900allows the splitter module 502 to split the input video into theplurality of segments including a set of frames.

At block 906, the method 900 includes determining the complexity of theplurality of segments by using separate complexity analyzer for eachsegment of the input video. The method 900 allows the complexityanalyzer module 604 including the plurality of complexity analyzers todetermine the complexity of the plurality of segments by using separatecomplexity analyzer for each segment. Here, each complexity analyzermeasures the complexity of every frame in each segment; thereby,determining the complexity of the segment. For example, consider aninput video is split into 3 segments i.e., Segment-1, Segment-2, andSegment-3. Unlike conventional systems, a complexity associated witheach segment (the Segment-1, Segment-2, and Segment-3) is analyzed andaccordingly bits for each segment are computed to generate a uniformvideo quality. Here, the complexity analyzer module includes pluralityof complexity analyzers i.e. complexity analyzer-1, complexityanalyzer-2, and complexity analyzer-3. The complexity analyzer-1determines the complexity of the segment-1 of the input video. Thecomplexity analyzer-2 determines the complexity of the segment-2 of theinput video. The complexity analyzer-3 determines the complexity of thesegment-3 of the input video.

At block 908, the method 900 includes determining the number of bits forallocating to each segment of the input video. The method 900 allows thecomplexity analyzer module 604 to determine the bits for allocating toeach segment of the input video. At 910, the method 900 includesencoding the plurality of segments and allocates the bits to eachsegment based on the complexity of the segment by comparing with theplurality of segments. Based on the complexity of each segment, themethod 900 allows the encoder module 506 to allocate bits to eachsegment by comparing with the plurality of segments received from thesplitter module 502. Here, the encoder module 506 will use the segmentinformation to make good local decisions within the segment. The encodermodule 506 accepts the information from the complexity analyzer module504 and accordingly allocates bits to each segment to encode the eachsegment into a different format. For each segment the encoder willachieve an average bit rate for the segment, and can allocate bits tothe high and low complexity sections of the video.

In some embodiments, a separate encoder can be used to receive eachsegment of the video from the complexity analyzer for transferring thesegment from one format to another format while achieving a desiredvideo quality. For example, consider the encoder module 506 consists anencoder-1, an encoder-2, an encoder-3 receives three segments namely,segment-1, segment-2, and segment-3 respectively from the complexityanalyzers. The encoder-1 allocates bits determined by the complexityanalyzer module 604 to the segment-1 by taking the inputs from thesegment-1. The encoder-2 allocates bits determined by the complexityanalyzer module 604 to the segment-2 by taking the inputs from thecomplexity analyzer-2 and the segment-2. The encoder-3 allocates bitsdetermined by the complexity analyzer module 604 to the segment-3 bytaking the inputs from the complexity analyzer-2 and the segment-3.Unlike conventional systems, different encoder's uses variable bit ratesencoding each segment of the video such as to generate a uniform qualityof video. Each bit rate for the segment is computed based on thecomplexity analysis of the segment.

Further, at 912, the method 900 includes combining the plurality ofsegments to form a single output video. The method 900 allows combinermodule 508 to combine the plurality of segments to form the singleoutput video. For example, the segment-1 from the encoder-1, thesegment-2 from the encoder-2, and the segment-3 from the encoder-3 arecombined to form the single output. At 912, the method 900 includesobtaining an encoded output video from the combiner module 208. Unlikeconventional systems, at 914, distributed video encoding system obtainsthe encoded output video while maintaining a desired quality by usingvariable bit rates for each segment instead of a globally optimized bitrate. Each segment is sent to a separate complexity analyzer, whichproduces the output file that describes the video complexity for eachframe. As the separate complexity analyzer is placed for each segment,the performance of the system is improved and the bits can be allocatedto the segments more accurately and quickly. Further, the complexity ofthe input video in the distributed video encoding is measured and bitsper frame to be allocated are determined for each segment of the inputvideo. The information about the per-frame bit allocation is passed ontothe encoder for allocating bits to each segment; thereby, obtaining goodaverage video quality.

FIG. 10 is a flow diagram illustrating a method 1000 for allocating bitsin a distributed video encoding using a separate complexity analyzer foreach segment of a video and adding a pathway that passes each segmentcomplexity measurement to an encoder, according to the embodiments asdisclosed herein. At block 1002, the method 1000 includes receiving aninput video for allocating bits for each segment based on the complexityof the segment. At block 1004, the method 1000 includes splitting theinput video into the plurality of segments. The method 1000 allows thesplitter module 202 to split the input video into the plurality ofsegments including a set of frames.

At block 1006, the method 1000 includes determining the complexity ofthe plurality of segments by using separate complexity analyzer for eachsegment of the input video. The method 1000 allows the complexityanalyzer module 604 including the plurality of complexity analyzers todetermine the complexity of the plurality of segments by using separatecomplexity analyzer for each segment. Here, each complexity analyzermeasures the complexity of every frame in each segment, and therebydetermining the complexity of the segment. For example, the input videois split into 3 segments i.e. Segment-1, Segment-2, and Segment-3. Inanother example, the input video can be split into N segments (i.e.Segment-1 to Segment-N). Unlike conventional systems, a complexityanalyzer module includes the plurality of complexity associated witheach segment (the segment-1, segment-2, and segment-3) is analyzed andaccordingly a per frame bit allocation for each segment is computed togenerate a uniform video quality. Here, the complexity analyzer moduleincludes plurality of complexity analyzers i.e. complexity analyzer-1,complexity analyzer-2, and complexity analyzer-3. The complexityanalyzer-1 determines the complexity of the segment-1 of the inputvideo. The complexity analyzer-2 determines the complexity of thesegment-2 of the input video. The Complexity analyzer-3 determines thecomplexity of the segment-3 of the input video.

At block 1008, the method 1000 includes determining the per frame bitbudget for allocating to each segment of the input video. The method1000 allows the complexity analyzer module 604 to determine bits forallocating to each segment of the input video. At block 1010, the method1000 includes encoding the plurality of segments and allocates bits toeach segment based on the complexity of the segment. Based on thecomplexity of each segment, the method 500 allows the encoder module 506to encode the plurality of segments and allocates bits to each segment.Here, the encoder module 506 will use the segment information and alsosegment complexity information computed earlier to make good localdecisions within the segment. The encoder module 506 accepts theinformation from the complexity analyzer module 604 and accordinglyallocates bits to each segment to encode the each segment into adifferent format. For each segment the encoder will achieve an averagebit rate for the segment, and can allocate bits to the high and lowcomplexity sections of the video.

In an embodiment, a separate encoder can be used to receive each segmentof the video from the complexity analyzer for transferring the segmentfrom one format to another format while achieving a desired videoquality. For example, consider the encoder module 506 includes anencoder-1, encoder-2, and encoder-3 receives three segments namely,segment-1, segment-2, and segment-3 respectively from the complexityanalyzers. The encoder-1 allocates the bits per frame budget determinedby the complexity analyzer module 604 to the segment-1 by taking theinputs from the complexity analyzer-1 (i.e. complexity information ofthe segment-1) and segment-1. The encoder-2 allocates the bits per framebudget determined by the complexity analyzer module 604 to the segment-2by taking the inputs from the complexity analyzer-2 (i.e. complexityinformation of the segment-2) and segment-2. The encoder-3 allocates thebits per frame budget determined by the complexity analyzer module 604to the segment-3 by taking the inputs from the complexity analyzer-3(i.e., complexity information of the segment-3) and segment-3. Unlikeconventional systems, different encoder's uses variable bit ratesencoding each segment of the video such as to generate a uniform qualityof video. Each bit rate for the segment is computed based on thecomplexity analysis of the segment.

Further, at block 1012, the method 1000 includes combining the pluralityof segments to form a single output video. The method 1000 allowscombiner module 508 to combine the plurality of segments to form thesingle output video. For example, the segment-1 from the encoder-1, thesegment-2 from the encoder-2, and the segment-3 from the encoder-3 arecombined to form the single output. At block 1014, the method 1000includes obtaining a encoded output video from the combiner module 208.Unlike conventional systems or other embodiments of the present system,each encoder uses the segment complexity information computed earlier tomake good local decisions within the segment and works faster, moreaccurately for allocating bits to the segments of the video; thereby,obtaining best overall video quality. The encoder assigns bit ratesacross all the segments (which may or may not be the same for eachsegment) of the video to obtain uniform quality.

As discussed with respect to FIG. 11, video is often uploaded to a videosharing site like YouTube, Vimeo, or Facebook by a user with anetwork-connected user device such as a wireless phone or a desktopcomputer. The user may be an amateur user, wishing to share video withfriends and family, or may be a professional videographer or a studio ora news-gathering and distribution service.

User devices are typically equipped with video encoders to reduce thesize of captured video. The encoding may be performed by running aprocess on a central processor (CPU) with or without assistance fromspecialized hardware circuitry.

Video quality, video bitrate, and CPU encoding time are all related. Fora given video encoder, typically for a constant video quality, a lowervideo bitrate can be achieved by increasing the CPU encoding time, e.g.,by using higher complexity encoding settings. If the CPU encoding timebudget per frame is held constant, then increasing video qualityrequires a higher video bitrate.

Some video encoders, generally called software encoders, e.g., videoencoder programs that run on a CPU, have settings that adjust how muchtime the CPU spends on each frame.

Some video encoders, generally called hardware encoders, also havesettings that control how fast the video encoding runs and how well thecompression occurs. Examples of hardware encoders include GPU basedencoders or encoders that use an application-specific integrated circuit(ASIC), e.g., in a mobile phone or a camera.

Video cameras (phones, DSLRs, video cameras, GoPro™) normally capturevideo at a very high quality and high resolution. In order to maximizequality and reduce the processing load on the consumer device, thedevices usually perform only a light encode, which means that the highvideo quality and resolution requires a very high bitrate. Such devicestherefore generate a lot of bits for captured video.

Video sharing sites (e.g., YouTube or Vimeo or Facebook, etc.) oftenlimit their bitrate on videos that are displayed. In order to limit thisbitrate, these sites lower the video resolution and lower the quality sothat the video fits within the bitrate. This video quality is normallymuch lower than what was originally captured by the camera.

Many video sharing sites will lower the video quality before displayingthe video to users. It may thus be wasteful to send the high qualityvideo up to the video sharing site because higher quality videostypically require more bandwidth and may require a longer upload time.In one advantageous aspect, the disclosed embodiments can encode videosat the client side to match the video quality that the video sharingsite will use.

Since video quality, video bitrate, and CPU cycles are often related andcontrolled by the encoding settings, the encoding at the user device canbe controlled to match available upload resources, e.g., bandwidth, CPUpower and battery life. Every client device will have a different amountof upload bandwidth available (e.g., depending on the network, such asWi-Fi, cell, fixed) and will have a different amount of CPU poweravailable (high end servers, low end desktops, laptops, tablets orphones).

FIG. 12 illustrates an example graph 1201 of trade-offs possible inencoding digital video for a given output bitrate of an encoder. Thecurve 1202 may represent the operation of a software-based videoencoding algorithm, e.g., implemented mainly using a CPU of a userdevice, with or without hardware assistance for various processing suchas transform computation and run-length or arithmetic encoding. Thecurve 1204 may represent the operation of a graphics-processor (GPU)based video encoding algorithm, e.g., implemented mainly using theparallelized pipeline of encoding, as is found in prevalent GPUs, Thecurves 1202, 1204 are plotted with the horizontal axis 1206 representinga measure of complexity of encoding, e.g., based on the amount of timetaken for per-frame encoding and the vertical axis 1208 representing theresulting video bitrate of the resulting video.

As can be seen from curves 1202, 1204, in general for a given encodingimplementation, e.g., GPU-based or CPU-based, the more time an encoderspends on the encoding, the better the quality of the resulting encodedvideo tends to be. As the amount of time spent per frame increases,however, benefit of encoded video size reduction may taper off for GPUbased encoding.

In some disclosed embodiments, the target output video quality is set toa fixed level, and then encoding settings are continuously adjusted suchthat upload bandwidth is maximally consumed and available CPU or GPUcapacity is maximally consumed. This ensures that the upload occurs asquickly as possible. On a device with low-powered CPU or GPU and a highbandwidth connection, an embodiment may only lightly encode the uploadedvideo because if video is encoded more diligently, then it would taketoo long for the CPU or GPU to encode the video and the upload bandwidthwould not be fully used, and the upload would take longer.

On a powerful device with a low bandwidth Internet connection, anembodiment may apply strong encoding to use as much CPU power aspossible and compress the video as much as possible to fit through thelimited upload bandwidth.

In both these cases, the video quality may be set to a constant value tomatch the optimal or desired settings of the video sharing site.

In FIG. 12, graph 1250 illustrates the difference between video bitrate,shown by curve 1254 and the encoding bitrate, represented by curve 1252.In the graph 1250, the horizontal axis 1256 represents the encodingeffort or processor resources spent and the vertical axis 1258 shows theresulting bitrate. Operational point 1260 represents a case where videoencoding may be performed, e.g., at ten times the real time rate (e.g.,300 frames encoded in one second for a 30 frames per second playbackrate) as shown by the fact that the encoding bitrate curve 1252 is muchhigher at operational point 1260 than the video bitrate curve 1254. Thevideo bitrate of an encoder may represent the number of encoding bitsper second used for encoding the video. For example, when the videobitrate of an encoded video is 1 Mbps, then approximately 1 million bitsfrom the encoded video may be used by a decoder every second duringdecoding and playback of the video. By contrast, if an encoder isoperating at 10× speed, then the encoder generates the 1 Mbps video atan encoder rate of 10 Mbps. During operation of a video encoder, theencoder bitrate and the video bitrate may be related by the equation:encoder bitrate=video bitrate x encoding speed. Thus, when the encoderis operating at 10× real-time speed, then the encoder bitrate is tentimes more than the video bitrate. The encoder bitrate may non-linearlychange as a function of the CPU time setting during operation, e.g.,based on how much time is spent in CPU processing for encoding eachframe. The intersection between the two curves 1252, 1254 may representthe operational point when video encoding is being performed in realtime such that the encoding rate and the video bitrate are equal. Thegraph 1250 illustrates a difference between video bitrate, e.g., thesize (in bits) of a video, divided by the playback duration of thevideo, and encoder bitrate, which is, e.g., the rate at which an encoderis outputting encoded bitstream.

Example Single Stream Mode Embodiments

In Single Stream mode, the input video is run through a single videoencoder. The output of the video encoder is buffered, and this buffer isuploaded to the Internet (uploader process). A rate control modulemonitors the size of the outbound buffer—how much has been encoded butnot uploaded yet—and uses this size to control the strength of theencoding. The system may set a high mark and a low mark for this buffer.When the buffer hits the high mark, it means that the buffer is fillingup: this means that the encoder is running faster than the uploadprocess because there is not enough upload bandwidth, so the encodingsettings are increased to tell the encoder to spend more time on eachframe to make the frame smaller. When the low mark of the buffer is hit,this means that the encoder is running slower than the uploadprocess—there is extra upload bandwidth available. In this case, theencoding settings are lowered to tell the encoder to spend less time oneach frame.

Example Multi-Stream Mode Embodiments

In this mode, the input video is segmented and run through multiplevideo encoders. The output of each video encoder is connected to anuploader process that pushes the content up to the video sharing site.Using multiple streams has two main benefits: 1. by using multipleparallel connections, higher upload throughput can often be achieved,and 2. some hardware and software encoders have higher throughput whenmultiple streams run through them. The algorithm for adjusting theencoding settings is similar to the above approach: the buffer size iscalculated using the sum of all of the output buffers, and a high andlow mark is established in the same way as above.

FIG. 13 is an example timing diagram 1301 of a video encoding processduring uploading of video from a user device (e.g., 1106) to a server1110. As depicted along the time line 1308, a first number of videosegments that make up the entire video to be uploaded may be encodedusing a default encoding setting (e.g., time per frame). In the depictedexample, two video segments 1302 and 1304 are initially encoded anduploaded to the server 1110 using a default encoding setting, which mayresult in, e.g., 1 Mbit/s video bitrate.

After the first number of segments are encoded and transferred at thedefault time per frame setting, a rate control module, which may beimplemented at the server 1110 and/or on the user device 1106, makes adetermination about the encoding setting (e.g., time per frame) that isto be used for encoding the next video segment. For example, with thedefault encoding settings, the video bitrate may be 1 Mbps. With theseencoding settings, however, the encoder may be running at 10× real-time(e.g., at operational point 1260), such that the encoder is generatingdata at an output rate of 10 Mbps. After the first two segments, whichmay be 6 seconds long each, are encoded, the rate control module mayobserve that the encoder data rate was 10 Mbps and the uploaded datarate was 5 Mbps. The outbound video buffer on the user device in whichthe encoded video is stored thus may be increasing in occupancy duringthe encoding and uploading of the first two segments due to the mismatchbetween the encoder rate and the available upload bandwidth.

The rate control module may then perform an encoding rate correction.Noting that buffer occupancy is increasing and the actual encoding isoccurring faster than target encoding rate, the rate control modulemight instruct the encoding process to spend more CPU or GPU time oneach frame and therefore reduce the rate at which encoded frames aregenerated. The change in encoding rate may be reflected in thesubsequent video segments being uploaded, e.g., segment 1306.

The rate control module may similarly control monitoring of the outboundvideo buffer to perform periodic correction to the encoded video bitrateby controlling various encoding parameters, e.g., the motion searchwindow, the target bitrate, the number of encoding options consideredfor encoding each macroblock of video, etc. The use, or lack of use, ofthese encoding parameters may cause the encoding speed to go up or down.For example, when using a CPU-based encoder, using a larger motionsearch window may slow down encoding process because more computationshave to be performed per macroblock, where a macroblock represents apre-defined pixel array, e.g., 16×16 pixels or video frames.

By monitoring operational parameters of the uploading process, the ratecontrol module can thus control the rate at which the encoder generatesthe encoded video bitstream such that the outbound buffer occupancystays within an upper or a lower margin. Further, by ensuring that theoutbound buffer is never empty, the rate control module may thusfacilitate the uploading of video at maximum possible rate at all times.

The above-outlined uploading process may be modified in variousembodiments using features such as user control or service providercontrol of the quality of uploading, the use of different operationalparameters, e.g., battery power left in a battery-operated device, thetariff a user may incur for bytes being uploaded, and so on.

FIG. 14 is flow diagram illustrating a method 1400 for uploading encodedor compressed digital video bitstream from a user device to a networkserver.

The method 1400 includes, at block 1402, determining an operatingsetting for uploading a video. The determination of operating settingsmay be made at a user device based on operating systems received from aserver in the network. For example, a service provider operating theserver to which the video is being uploaded may provide a minimum and/ora maximum acceptable quality for the video being uploaded. Alternativelyor in addition, the operating settings may be determined by presenting,on a user interface, a menu of operating setting selection and byreceiving a selection of the operating setting from the user interface.In some embodiments, the menu of operating setting selection includes anencoded video quality selection menu.

In some embodiments, the user device may derive the default encodingsetting from the operating setting. The encoding settings may include,e.g., an output video bitrate for encoding the video, number of B or Pframes (Bidirectional or Predictive frames), and so on. In someembodiments, the default encoding settings may be selected as theencoding settings that were used when the last segment of a previouslyuploaded video was encoded. In some embodiments, the default encodingsettings may be based on the current network connectivity of the userdevice. For example, a higher bitrate, e.g., 5 Mbps, may be used forencoding when the user device is connected to broadband internet and alower bitrate, e.g., 1 Mbps, may be used when the device is connectedwith the server using a wide area network connection such as a 3G or a4G connection.

The method 1400 includes, at block 1404, transmitting a first set ofsegments using a default encode setting. For example, as described inthis document, in some embodiments, the first set of segments mayinclude first two GOPs of a video.

As described in this document, after the first few segments aretransmitted by the user device and received by the server, the servermay be able to determine whether the user device is encoding video at afaster-than-transmission rate or a slower-than-transmission rate. Forexample, the knowledge of the occupancy of outbound buffer may be takeninto consideration. The rate control module described herein may thusdetermine revised encoding settings.

The method 1400 includes, at block 1406, receiving a revised encodersetting that is based on an optimal uploading setting. For example, therevised encoder setting may be lower limited by a minimum time spent perframe and upper limited by a maximum time spent per frame.

The method 1400 includes, at block 1408, encoding video using therevised encoder setting. The encoding may use CPU and/or GPU forencoding, as controlled by the encoding setting. In some embodiments, tocontinually ensure that the encoded video bytes are available fortransfer over the network interface, the CPU (or GPU) of the user devicemay be used to its maximum possible availability during the time videoencoding and uploading is being performed. As described herein, otheroptimal setting criteria, e.g., maximum battery saving etc. may be usedfor encoding also.

Example Uploading Quality Setting

In some embodiments, the network server to which the client device wantsto upload the video may specify quality of video that is acceptable forupload. The network server may specify an upper limit on the quality ofvideo, e.g., based on a service layer agreement (SLA). Additionally oralternatively, the network server may specify an acceptable lowerthreshold for quality below which video will not be accepted by theserver. The quality setting may take place every time a user wants toupload a video. Alternatively or additionally, the network server mayspecify quality setting on a periodic basis, depending on the user's useof her storage space at the network server. Alternatively oradditionally, the network server may specify a different quality settingevery time there is a change in the network over which the user deviceis accessing the network server. For example, the network server mayspecify the use of one quality setting when the user device is operatingwith a broadband Internet (Wi-Fi) connection and another quality settingwhen the user device is operating in a 3G or 4G network.

Alternatively or additionally, a user may be able to specify or requesta quality setting prior to uploading a video. For example, a userinterface may be presented to the user at the beginning of theuploading, requesting which one of a number of quality settings (e.g.,low, medium or high) does the user want to use for the uploading of thevideo.

Example Optimal Criterion for Uploading

Users (and service providers) may want to use different optimal criteriato control the uploading process. The criteria may include, e.g., uploadtime, total number of bits uploaded, upload quality, and so on.

In some embodiments, the above techniques may be implemented in a userdevice that includes at least one processor, at least one memorycomprising instructions configured to cause the at least one processorto perform the method 1400 described herein.

During the encoding, an encoded bitstream may have to be generated toensure that a key frame occurs every given number of seconds. Forexample, in some implementations that use hypertext transfer protocolbased live streaming (HLS), a key frame may be expected by a videoplayer at some frequency, for example, at least once every 10 seconds.However, the original video available from the user device may notfollow this periodicity of key frames. For example, video captured froma user device may use a 6-second GOP, while the video sharing serviceprovider may want to store and distribute video sequences at a differentGOP length, e.g. 10 seconds apart.

FIG. 15 depicts an example workflow in a distributed video encodingsystem 1500. The system 1500 may be operated by a video uploading andsharing service provider. Distributed video encoding involves splittingthe input videos into segments and passing the segments to a cluster ofworker nodes. The worker nodes independently encode the videos and shipthe output segments back to the master node. The master node thenstitches all the segments together to create a final output.

At block 1501, an input video sequence is received by a master node froma user device 1106, 1116. The input video sequence may be a compressedvideo sequence or an uncompressed video sequence. At block 1502, themaster node may split the video into multiple segments. The master nodemay then assign segments to different worker nodes for performing videoencoding or encoding. At 1503, each worker node encodes the segmentreceived. At 1504, each worker node sends back the resulting encodedbitstream to the master node. At 1505, the master node combined thesegments back to generated encoded bitstream for the video. At 1506,master node then stitches all the segments together to create a finaloutput. A similar workflow of splitting, encoding, and combining is alsodescribed with respect to FIG. 3, FIG. 5 and FIG. 6.

This technique may be sufficient when the video has intermittent IDR(Instantaneous Decoder Refresh) frames, which are key frames (Intraencoded, or I, frames is another example of key frames), since thesplitter can look for IDR frames to segment the input video file. Thisis called a “closed GOP” type bitstream, where the frames inside a GOPrefer to other frames in the same GOP. Since an IDR is an anchor frameany future frames cannot refer to a frame earlier than the anchor frame.Hence, the sanctity of each video segment is preserved. Thereby theencoder (worker) nodes can independently process each chunk.

FIG. 16 illustrates an example of a video 1600 when the input video is aclosed GOP sequence.

As depicted in FIG. 16, all the references, indicated by connectors1602, 1604, are within the GOP structure. So in this case, the videosegmenter creates segments or chunks at the end of frame P4 and createssegments as I1B2B3P4 and I5B6B7P8 and so on. Each of these segments isindependently processed by the worker nodes because there is no crossGOP referencing.

FIG. 17 depicts an example of an open GOP video 1700. The illustratedexample shows that the frame P4 is predicted from the frame I1, theframe B2 is bi-directionally predicted using references from I1 and I5and the frame B3 is bi-directionally predicted from the reference framesI1 and I5.

When encoding an open GOP video, the above described chunking strategydoes not work where there are references outside of GOP boundaries. Inthis case, just segmenting at an I-frame boundary would cause theresulting segments to fail during encoding, because the dependencies arenot met, resulting in bad video quality.

As seen from FIG. 17, if the segmenter cuts at I-frame boundaries, theworker nodes would fail encoding because B2 has dependency on I5(depicted by connector 1702), which falls in a different segment thanB2.

In one aspect, an intelligent segmenter described herein preservessegment level sanctity of open GOP video frames. In case of an open GOPvideo, the segmenter includes a few extra frames at the GOP boundary toaccount for cross GOP referencing. If a previous IDR frame, I frame or akey frame, exists then the segmenter can choose to include all theprevious GOPs until that key frame. Alternatively, the segmenter couldjust include the maximum reference length of frames for that videocodec. Every codec has a maximum length beyond which it generally cannotrefer a frame for motion estimation/compensation. This leads to multiplesegments with over lapping frames. In order for these segments to beencoded on the worker nodes, additional header information is required,which is copied over from the previous key frame.

In order to not duplicate the overlapping frames while encoding,metadata can be sent along with the file name to the worker nodes. Themetadata contains the number of initial frames to be skipped before theencoder starts encoding. At the end of encoding the worker node submitsthe encoded segments back to master, which then merges them to create afinal output.

In some embodiments, each encoder in a distributed encoding system maybe provided with contiguous video frames in excess of the frames thatmake up a video segment that the encoder is asked to encode. Forexample, a first encoder may be given frames 1 to 12 of a video, butwill be asked to encode frames 1 to 10. This way, regardless of the GOPdependency, the worker node is assured that all possible video framesfrom which motion vectors are derived in the input video are availableto the worked node for encoding. At the same time, because a sufficientnumber of video frames are available and are of duration longer that thedesired key-frame insertion period (e.g., 10 frames), the worker nodecan perform encoding to precisely insert a key frame where desired.

FIG. 18 illustrates how worker nodes can perform encoding. Withreference again to the video 1700 which may be input to the encodingprocess, the operation of segmentation based encoded may be performed asfollows. In some embodiments, prior to the segmentation operation, themaster node may generate a dependency map. The dependency map mayidentify, for each frame in a segment, other frames on which decoding ofthat frame depends, e.g., the reference frame(s) used in encoding thatframe. Based on the dependency map, the intelligent segmenter mayinclude in each segment frames of two types—frames that are to beencoded by worker nodes, and additional frames that are included in thesegment because the frames to be encoded depend on the additionalframes. For example, in 1800, a first worker node may start encoding atI1 (first frame of received segment 1), and stop encoding at therightmost P4 frame (last frame of received segment 1), as indicated bythe metadata. However, because frames B2 and B3 are bi-directionallypredicted from the frame I5, the intelligent segmenter includes theframe 15 in Segment 1. In other words, the intelligent segmenter mayinclude, in some segments, additional video frames that are not to beencoded in the output video, but are used only for encoding other videoframes in the segment.

Segment 2, which is the next segment temporally after Segment 1,represented in 1802, represents the case when all frames in the segmentare sent to a worker node and are also to be encoded by the worker node,e.g., as may be the case for a closed GOP segment. Based on thedependency map, the intelligent segmenter may thus send some frames,e.g., frame I5 in multiple segments, e.g., both in Segment 1 (but notencoded by the worker node that receives Segment 1) and in Segment 2,where it will be encoded by the worker node that receives Segment 2.

FIG. 19 illustrates an example method 1900 of encoding a compressedvideo bitstream. The method 1900 may be implemented on the network-sideof a social media network or a video sharing service provider's network.

The method 1900 includes, at block 1902, receiving a video. The videomay be a compressed video bitstream. In the received video, key framesmay occur nominally a first time duration apart, wherein a key frame isencoded without depending on another frame in the compressed videobitstream. The time duration may be specified in seconds, frame numbers,and so on. For example, in some embodiments, the time duration may be 10seconds to 20 seconds and may correspond to an intended streamingprotocol used when allowing other users to access the video bitstream.

The method 1900 includes, at block 1904, building a dependency map thatidentifies the reference frames used for each encoded video frame. Someexamples of the dependency map are illustrated in FIG. 17 and FIG. 18.At block 1906, the received video is segmented using the dependency map.For example, as discussed with respect to FIG. 17 and FIG. 18, eachsegment may include frames that are encoded using only other referenceframes within the same segment. The method 1900 includes, at block 1908,providing, or distributing, segments of the compressed video bitstreamrepresenting corresponding to a second nominal time duration to multipleencoding nodes, wherein each segment includes at least one key frame,and wherein at least some segments include video frames that occurtemporally before and after the at least one key frame in a displayorder.

The method 1900 includes, at block 1910, receiving back results ofencoding performed by each encoding nodes. The received encoded segmentinclude the corresponding segment of compressed video bitstream suchthat key frames in resulting encoded video bitstream occur a second timeduration apart. The method 1900 may include, at block 1912, combining,or concatenating, the received encoded segments into a single bitstreamrepresenting the encoded representation of the received video.

In some embodiments, the encoding includes encoding by changing temporallocation of a key frame in resulting encoded video bitstream.

In some embodiments, the method 1900 includes determining, for eachframe, in the display order, most distant past and future frames fromwhich motion vectors for each frame are formed. In some embodiments, thedetermining is performed using motion vector information from thecompressed video bitstream. In some embodiments, the determining isperformed without uncompressing the compressed video bitstream.

In some embodiments, the encoding by each node is performed on fewerthan all frames making up the corresponding segment. For example, asdiscussed herein, some frames before and/or some frames after key framesmay be used for the sake of continuity of motion vectors only and may bedropped from the encoding process.

In some embodiments, the method 1900 further includes providing, to eachof the multiple encoding nodes, information regarding location of a keyframe in the encoding operation. In some embodiments, the method furtherincludes encoding, by one of the multiple encoding nodes, the at leastone key frame in received video segment to become a non-key frame inresulting encoded video bitstream.

Video encoders normally have many settings that adjust specificattributes of the encoding process. The effect of each of these settingscan depend on the input video—some of the settings might make an encodeworse for a certain input video, e.g., high motion, but will make adifferent input video, e.g., low motion better.

FIG. 20 depicts a graph 2000 to highlight this observed behavior of thevideo encoders at a given bitrate. The horizontal axis 2002 representsan encoding parameter setting, e.g., the amount of time spent by theencoder to encode each frame, and the vertical axis 2004 representsquality of output compressed video for multiple video encoders (multiplecurves 2006, representing multiple encoder parameter settings 1, 2, . .. Z, etc., where Z is an integer). The actual variations may depend fromfeature to feature, but generally, video encoders show varying qualityoutputs based on how the encoder setting.

It is often difficult to select a priori the best settings and, inpractice, encoder settings are often selected based onoperator-experience, or are pre-selected without attempting to customizevideo encoding settings to individual video content.

Some examples of video encoding settings can include: frame distancebetween key frames, how many predictive or bi-directional framesintervening key frames to use, threshold used for detecting scenechanges by comparing two successive video frames, whether or not toperform intra-frame motion prediction, whether or not to use differentquantization matrices, which of the multiple coding options to use forcoding bits (e.g., variable length encoding or arithmetic encoding),whether or not to use fading detection, the motion search window to beused for each video frame, and so on. Some of these settings may havediscrete (e.g., on/off or high/medium/low) values, while others may havecontinuous values. This is just an example list and one of skill in theart would appreciate that there are hundreds of design trade-off optionsthat can be made in selecting a particular mode in which to run a videoencoder, with each design option being a good choice for one type ofvideo but not a good choice, in terms of compression efficiency, foranother type of video.

In one example aspect, the realization that different settings areoptimal for different types of video, and that a single video might havemultiple different segments of video for which different settings areoptimal, is used to achieve video compression ratios higher than thatcan be achieved by a given single video encoding setting on amulti-segment video. The present document discloses, among othertechniques, a mechanism to find a relatively optimal setting (e.g. froma set of thousands of settings) for each different segment, and thenassemble all of the encoded results into a single output video.

In some embodiments, a set of test video clips, or segments, may beencoded through an automated encoder engine. The encoder engine mayencode each test clips with hundreds or thousands of different settingsto understand the effect of each video setting on each test clip. Theresults of these encodings are used to train a training module that candetermine optical encoding configurations for future video encodings.

In some embodiments, the training phase may be implemented using aneural network that uses a learning algorithm in which a cost criterionsuch as rate-distortion or visual rating score is used for training theencoding parameter learning.

FIG. 21 illustrates an example arrangement of video encoders 2100 toproduce video encoding or encoding with very high compression ratios,while maintaining quality. In some embodiments, a library of test videos2102 may be used as input to an automated tester module 2104 whichcycles through all test sequences using all possible (could be inthousands) combinations of video encoding parameters to generate adatabase 2106 of results. This database 2108 may be used to train amachine learning algorithm such as a neural network settings generator2116.

The arrangement 2100 may receive an input video 2108 for encoding orencoding, e.g., from a user upload.

In some embodiments, an intelligent segmenter 2110 may divide the inputvideo into different segments 2114. This intelligent segmenter may learnan optimal place to segment the video and place output key frames tomaximize quality by detecting scene change.

In some embodiments, a fingerprint generator 2112 may measure variousvideo characteristics of each segment. The fingerprint generator 2112may thus identify a “fingerprint” of a video, e.g., the qualities of thevideo that make it more or less amenable to certain configuration ofencoding parameters. The fingerprint generator may feed thesecharacteristics to a neural network settings generator 2116.

In some embodiments, the neural network settings generator 2116 may beprovided with input video segment characteristics and may determine aset of video encoding settings that should include a setting for arelatively optimal encoding, e.g., the smallest output file size for agiven quality level. The number of settings returned for each segmentmay be a configurable parameter: additional settings increases thechance of finding the optimal setting and improves compressionefficiency, while fewer settings results in a faster overall encode.

In various embodiments, various machine learning techniques may be usedto train the encoding process for how to pick the best video settingsfor each segment of a video to be encoded. For example, the complexityanalyzer 504, 604, could be used to identify and report complexity of asegment as an input to the fingerprinting process.

In some embodiments in which multiple encoders 2118 are used, each videosegment is encoded multiple times with the settings returned from theNeural Network Generator 2116.

After the video segments have been encoded, a segment selector 2120analyzes each segment and picks the smallest sized segment 2122 thatmeets a pre-determined quality requirement (e.g., a threshold quality).

The video segments are joined together into a single output video 2124.

As a result, each segment of the output video has been encoded withdifferent settings that are relatively optimal for that segment, whichmeans that as a whole, the output video meets the predetermined qualityand has the relatively smallest file size.

FIG. 22 illustrates an example flowchart for a method 2200 of encoding avideo.

The method 2200 includes, at block 2202, dividing a video into multiplesegments for encoding.

The method 2200 includes, at block 2204, deriving for each segment, acorresponding encoding fingerprint. In some embodiments, the derivingoperation includes analyzing a ratio of bit utilization by differentframe types in the videos, wherein the frame types include anintra-encoded frame type and an inter-encoded frame type. In someembodiments, the deriving operation includes analyzing a ratio of bitsused in the videos for encoding motion based blocks and bits used in thevideo sequence for encoding non-motion based blocks.

The method 2200 includes, at block 2206, determining, based on theencoding fingerprint, an encoder setting for each segment. In someembodiments, the determining the encoder setting includes searching adatabase of encoder settings for a best match with the encodingfingerprinting of each segment. In some embodiments, the database ofencoder settings is produced by encoding multiple test videos usingmultiple encoder settings and collecting quality measurements ofresulting videos. In some embodiments, more than one encoder setting maybe determined for each segment. Alternatively, or additionally, theparameter values for a given encoder setting may also be determined. Forexample, one encoder setting may relate to using a rectangular searchwindow while another encoder setting may relate to the sequence in whichmotion vector search is performed within the search window, and withinthe encoder setting, a parameter may specify the size of the searchwindow.

The method 2200 includes, at block 2208, encoding each segment using thedetermined encoder setting or settings. In some embodiments, theencoding of each segment is performed using distributed resources inwhich at least some of the encoding is performed by different encoders.The method 2200 includes, at block 2209, selecting from among thedifferent encoded bitstreams generated by using different video encodersettings, a bitstream that meets a certain criterion. The criterion maybe, e.g., a bitstream having a smallest size or a least number of keyframes.

The method 2200 includes, at block 2210, generating an encoded versionof the video by concatenating the selected encoded segments. Thegeneration of an encoded version of the video may thus be accomplishedsuch that each segment of the encoded version is selected to be a bestchoice according to a specified criterion, from among may other segmentsencoded using a variety of different ways of encoding the correspondingvideo portion.

In some embodiments, the method 2200 may further include encoding byassigning, to multiple worker nodes, segments of the video for encodingalong with encoding settings to be used; and receiving, from themultiple worker nodes, a resulting bitstream for each segment; andcollating together the received resulting bitstreams to produce theencoded version of the video.

As described in FIG. 21, e.g., in some embodiments all segments may beencoded concurrently, or at the same time, by submitting the segments tomultiple encoders for encoding.

It will be appreciated that the disclosed techniques can be used toproduce video that is segment-wise encoded using the best possibleencoder setting for each segment, without having to incur time penaltyby using distributed encoding resources. Furthermore, by using machinelearning, e.g., generator 2116, the encoding parameter selectionalgorithm can be continuously improved.

In various embodiments, techniques for improving image stabilizationusing distributed processing are disclosed. In a distributed videoencoding system, a video is encoded by splitting the video into videosegments and encoding the segments using multiple encoders. Prior tosegmenting the video for distributed video encoding, image stabilizationis performed on the video. For each frame in the video, a correspondingtransform operation is first computed based on an estimated cameramovement. Next, the video is segmented into multiple video segments andtheir corresponding per-frame transform information. The video segmentsare then distributed to multiple processing nodes that perform the imagestabilization of the corresponding video segment by applying thecorresponding transform. The results from all the stabilized videosegments are then stitched back together for further video encodingoperation.

With the mass proliferation of cell phones with camera, there is growthin videos being taken and shared. Non-professionally captured videos andvideos captured using hand-held devices such as cell phones and tabletcomputers, often suffer from lot of shakiness due to instability ofhuman hand and external factors like wind, etc. The resulting video maybe shaky, e.g., stationary objects may change position in successivevideo frames not from movement in the object but due to movement in thecapture device. The resulting video is not just unpleasant to watch butalso may suffer from poor video compression quality because bits may beexpended in capturing object movement where, in fact, there was notmovement of the object. It may be beneficial to process the video usingan image stabilization stage in which instability due to camera movementis detected and suppressed or eliminated.

In some embodiments, a two-step image stabilization process may be used.In the first pass, video frames may be analyzed to detect shakiness anda corrective transform or transforms may be computed to substantiallymitigate the effects of unstable camera. In the second pass, thetransforms are applied to the source video frames to produce stabilizedvideo.

The existing methods perform these passes linearly on a single machine,which is extremely slow. We disclose a method to accomplish imagestabilization in a distributed encoding environment.

FIG. 23 depicts an example of a video frame sequence 2300 in whichcamera shakiness has occurred. The sequence 2300 illustrates 6consecutive frames of a video captured by a camera when the camera wasshaking. With respect to the first captured frame F1, an upward shake isobserved at frames F2, F3 and F4. Frames F4, F5 and F6 represent adownward shake of the camera. The dashed line 2302 represents theapproximate path, or up-down movement, of the camera. While not depictedexplicitly in the drawings, a determination can also be made aboutcamera rotation, whether camera rotated clock-wise or anti-clockwise canalso be made.

In some embodiments, image stabilization may be performed as follows:

Pass 1: In this pass, camera motion is detected by figuring outdisplacements in the positions of some objects that occur in two or moretemporally sequential frames. An object from a frame is matched withsame object in the next frame(s) to determine Frame Motion, Mf. Theestimated frame motion Mf is then utilized to determine overlappingareas between adjacent frames. Each frame is then cropped and zoomed toensure that the maximum overlapping area persists between adjacentframes. This is accomplished by creating a transform T(n) for everyframe 0 to N (N+1 total frames) in the video. In some embodiments, alist with N+1 entries may be generated, with each entry in the listcorresponding to one of the T(n) transforms.

Pass 2: For each video frame n, the Transform T(n) is applied to thevideo frame. Subsequently, the video is encoded to improve imagestability.

In a conventional method, the operations of Pass 1 may be performed onan entire video and a list may be created for details of transforms T(n)performed for every step.

Next, the entire video is encoded, or compressed, by first applying theappropriate transform T(n) for each frame 0 to N to stabilize thecontents for the frame against camera movement.

Since this approach is linear, it may be extremely slow, making itundesirable in real world situations.

In some embodiments, distributed computing resources are used to achievedistributed image stabilization.

In some embodiments, the process of image stabilization is distributedamong multiple nodes for achieving better performance (e.g., end-to-endprocessing speed). For example, Pass 1 may be performed as describedabove on a single node/machine (master node) since it is data dependentacross adjacent frames. A transforms file T is created which containstransform T(n) to be applied for each frame 0 to N in the input video.

Next the input video is split (e.g., at key frames) into multiplesegments. For example, the N+1 frames may be divided into K segments,S(k), where k=0 to K−1. The number of frames in each segment S(k) may berepresented as L(k). In general, depending on key frames in the inputvideo, L(k) may have different values. The transforms file T (whichcontains the transforms T(n) corresponding to frames 0 to N of the inputvideo) is also split and re-indexed per the segment frame offsetsmatching the input video segments. The segments and correspondingsegment transforms file Ts(k) (created from the original transforms fileT) is sent to each worker node for processing. Each worker node appliesthe transforms Ts(k) in the received transform file Ts(k) to the framesof the segment (e.g., 0 to K frames in the segment) and generates animage-stabilized segment. As an example, in some embodiments, an inputvideo may comprise 3,000 video frames (N=2,999), which may be dividedinto 200 segments (K=200). The number of frames in each segment may bebetween 1 to 15 frames, and there may be 3 worker nodes for imagestabilization. The master node may send first 65 segments to the firstworker node, along with first 65 entries of the transform matrix list T,the next 65 segments and corresponding transform entries to the secondworker node, and the last 70 segments and the corresponding transformentries to the third worker node.

At the end of processing the master node collects all individualsegments and stitches them together, thereby creating the final imagestabilized video.

FIG. 24, FIG. 25, FIG. 26, FIG. 27 and FIG. 28 depict an operationsperformed by the master node and worker nodes.

At 2400, one of the nodes, called a master node, analyzes the frames ofthe video and generates a file T of transforms T(n).

At 2500, the master node splits the input video into multiple videosegments S(k). The master node computes a frame offset used for eachsplit and keeps track of these frame offsets to produce the segmenttransform file Ts.

At 2502, the master node splits the transforms file, T, into multiplesegment transforms Ts(k) using segment frame offsets calculated at 2500.

At 2600, the master node distributes the segments and correspondingtransform files to worker nodes. For example, first worker node mayreceive S1 segments, and T1 segment transforms, the nth worker node(where n may be between 1 and W, where W is the number of worker nodes)may receive Sn segments and corresponding Tn segment transforms from thetransform file T.

At 2700, the worker nodes apply the transforms and create stabilizedsegments. Upon completion of the stabilized segment, each worker nodesends each segment S1e, S2e, . . . up to Sne, where n is between 1 andW, to the master node.

At 2800, the master node stitches all the stabilized video segments backtogether.

FIG. 29 illustrates an example method 2900 of distributed imagedstabilization.

The method 2900 includes, at 2902, analyzing, using a master processor,frames of a video to generate a list of per-frame transform formitigating camera movement for each frame;

The method 2900 includes, at 2904, partitioning, using the firstprocessor, the video into multiple video segments. In some embodiments,the master processor and the first processor may be the same processor.

The method 2900 includes, at 2906, providing, from the first processorto a plurality of worker processors, one or more of the multiple videosegments and a corresponding segment transform file.

The method 2900 includes, at 2908, performing, by the plurality ofworker processors, image stabilization on the received video segment(s)by applying the corresponding segment transforms to the received videosegments to produce stabilized video segments.

The method 2900 includes, at 2910, conveying the stabilized videosegments from the plurality of worker processors to the masterprocessor.

The method 2900 includes, at 2912, generating (e.g., by stitching orconcatenating), by the master processor, a stabilized video from thestabilized video segments.

In some embodiments, analyzing frames includes analyzing motion vectorsassociated with the frames to identify a global component to the motionvectors. In some embodiments, the analysis is performed on a compressedrepresentation of the frames. In some embodiments, frame analysisfurther includes decompressing at least some of the frames at leastpartially.

In some embodiments, the frame analysis includes applying a rotationaltransform to at least some of the frames. In some embodiments, thetransform for mitigating camera movement includes compensating for atleast one of a left-right movement, an up-down movement and a rotationalmovement.

In some embodiments, generating the stabilized video from the stabilizedvideo segments includes smoothing camera movement across contiguousstabilized video segments.

In some embodiments, partitioning into multiple video segments includescreating each video segment by including a key frame at a given positionin the video segment.

As described before, distributed video encoding involves splitting theinput videos into segments and passing the segments to a cluster ofworker nodes. The worker nodes independently encode the videos and shipthe output segments back to the master node. The master node thenstitches all the segments together to create a final output.

FIG. 30 illustrates an example of a distributed video encoding system3000. At 3001, an input video is received by a master node from a userdevice 1106, 1116. The input video may be a compressed video or anuncompressed video. At 3002, the master node may split the video intomultiple segments. The master node may then assign segments to differentworker nodes for performing video encoding. At 3003, each worker nodeencodes the segment received. At 3004, each worker node sends back theresulting encoded bitstream to the master node. At 3005, the master nodecombines the segments back to generate an encoded bitstream for thevideo. At 3006, master node then stitches all the segments together tocreate a final output. A similar workflow of splitting, encoding, andcombining is also described with respect to FIG. 3, FIG. 5 and FIG. 6.

One aspect to streamline the workflow described in FIG. 30 is to ensurethat the worker nodes are highly utilized without being over-utilized.Over-burdened worker nodes can have a direct hit on performance andunderutilized nodes are a wasted resource of the distributed system.

Considering that encoding can be done on the CPU or GPU, these two maybe considered independent resources in a system. The quality of encodingon a CPU is different from quality of encoding on a GPU.

FIG. 31 depicts an example flowchart of a method 3100 implemented at themaster node. Each input video, Vin, is analyzed by the master node usinga classification scheme C (Vin, output) and determined if the video isto be encoded using CPU or GPU. Factors like target resolution andquality are considered in making this decision.

Vin-resource=C(Vin, output), calculated at 3102.

At 3104, the master node segments the video, The Vin-resource attributeis attached to each segment, Vs. Alternatively, individual segmentscould also be analyzed to determine if they would be ideal for GPU orCPU encoding. This yields true heterogeneous behavior, even with a file.At 3106, a determination is made about whether a given segment is forencoding by a CPU or by a GPU.

Segments Vs with a “CPU” attribute are queued in a GPU-queue (3108).

Segments Vs with a “CPU” attribute are queued in a CPU-queue (3110).

As depicted in the example flowchart of FIG. 32, each worker node run aprocess, P, 3200 that determines its available CPU and GPU resources(3202).

If the CPU resources of a node fall below a threshold (e.g., as checkedin 3204), new job (segment) is picked from a CPU queue. Then, CPU basedencoding is performed and encoded segment is returned to the master(3206).

As shown in FIG. 33, a similar process 3300 is performed at the workednode to determine GPU resource availability (3302). Similarly if the GPUresources of a node fall below a threshold, a new job (segment) ispicked from the GPU queue for processing (3304).

At the end of processing the worker nodes submit encoded chunks back tothe master node (3306).

Master node reassembles the individual segments and generates the outputfile.

FIG. 34 is a flow diagram illustrating an example method 3400 fordistributed video encoding. The method 3400 may be implemented at amaster server, as described herein.

The method 3400 includes, at block 3402, receiving a video for encoding.In some embodiments, e.g., the video may be received from a user devicethat uploads the video to the master server.

The method 3400 includes, at block 3404, establishing a pool of workerserver nodes for encoding the video. In some embodiments, theestablishing the pool of worker server nodes includes querying eachserver node, prior to the receiving the video, for information aboutencoding capabilities of the worker server node. In some embodiments,the encoding capabilities include at least one of protocol compliance, asoftware-based encoding capability, e.g., a CPU-based encoder, and agraphics processor based processing capability, e.g., a GPU encoder.

The method 3400 includes, at block 3406, splitting the video intomultiple segments. In some embodiments, the splitting includes splittingthe video such that each segment includes at least one key frame fromthe video that is independently decodable.

The method 3400 includes, at block 3408, submitting the multiplesegments to the pool of worker nodes for encoding based such that eachsegment is submitted to a corresponding worker node based on a run-timeavailability of the corresponding worker node. In some embodiments, therun-time availability includes determining whether the correspondingworker node is able to meet a target time within which a segment is tobe encoded. The method 3400 includes, at block 3409, instructing to eachworker node in the pool whether that worker node should use CPU-basedencoding or GPU-based encoding to encode the segment submitted to theworker node. In some embodiments, the master node may first decidewhether a segment should be encoded using CPU-based encoding orGPU-based encoding and based on this decision, the master node mayselect one of the available worker nodes from the pool of worker nodesthat can perform encoding using the selected encoding method (GPU-basedor CPU-based).

In some embodiments, the method 3400 further includes receiving encodingparameters for the video and wherein the establishing the pool includesselecting worker server nodes whose encoding capabilities are compatiblewith the encoding parameters.

In some embodiments, the method 3400 further includes receiving encodedsegments from the worker nodes, and reassembling the encoded segments toproduce encoded sequence. In some embodiments, the method 3400 includesperforming another encoding on the video sequence at a differentencoding bitrate to produce another encoded copy of the video sequence.In some embodiments, the method 3400 may further including making adecision about whether to use a CPU or a GPU for encoding segments ofthe video, e.g., based on resource availability.

When a distributed encoding system, e.g., as depicted in FIGS. 2, 3, 5,6, 7, 15, 30, etc. is used for encoding segments of multiple videosequence, the resulting encoding can finish at a very fast rate. Forexample, a 30 minute video may be split into 1 minute long segments,with the resulting 30 segments being fed to 30 worker nodes that canencode the video at 2× rate, thereby generating encoded bitstream forthe 30 minute video in slightly more than 30 seconds (depending on thenetwork traffic and splitting/combining overhead).

However, in video file upload and sharing networks, often, hundreds ofusers are uploading their video files for storage and sharing. Also,some users may have service level agreements with the service providerthat their video file, once uploaded, should become available forworldwide viewing within a specified time period. This time period maybe “real time” especially for news sources that want their news tobecome available for public viewing as soon as the news clip us uploadedto the server.

Such demands on the distributed video encoding system of having a largenumber of videos in a queue for encoding, while at the same time wantingto encode some videos at extremely quick turnaround, may poseconflicting demand on resources. For example, while the above-discussed30 minute video may be encoded in 30 seconds if all distributedresources were available to the video, this video may be queued behind500 videos from other users, which may be 2 minutes long each, therebycausing several minutes of delay before the encoding of the 30 minutevideo finished.

The techniques presented in this document can be used, in one aspect, toovercome this operational disadvantage by reprioritizing video encodingon a segment by segment basis instead of a video by video basis. Thisallows, for example, for encoding to be paused after a single segmentand resources to be redistributed or reallocated as desired, rather thanneeding to wait for the end of encoding of an entire video, as requiredby traditional encoding systems.

Distributed video encoding can achieve must faster compression than aserial operation of a single encoder since a master node distributes thetask of encoding among multiple worker nodes. If Ms is the time takenfor encoding one video on a single machine, the following equationyields the time taken on a distributed environment.Md=(Ms/n)+O, where

Md=total time taken to process a video on a distributed encodingplatform;

N=number of worker nodes; and

O=overhead for splitting, merging and job distribution. This is linearto the length of the video.

It is beneficial to some users to have an arrangement with a videosharing service provider to be able tm meet a fixed speed of encodingirrespective of number of resources available or any other users'encoding jobs undertaken by the service provider. The disclosedtechniques can be used to monitor and control speed of each encode via afeedback loop from worker nodes to the master node.

FIG. 35 depicts an example distributed video encoding system 3500 inwhich a master node 3501 performs the task of splitting a video intomultiple video segments of smaller sizes and dispatching them tomultiple worker nodes as encoding jobs, as described in the presentdocument.

A job submitter component, residing in master node, may be responsiblefor submitting segment level jobs to a job_queue which holds a queue ofsegments to be sent to worker nodes.

In a traditional case, the job_submitter may keep pushing each videosegment as it arrives into the job_queue and the worker nodes pull jobsfrom the job queue, process them and send them back to the master node.Assuming, an input video I is split into “n” segments thereby creating njobs. Ij1, Ij2, . . . , Ijn. Similarly a second input video K would haveKj1, Kj2, . . . Kjm jobs. In the traditional case, because of thequeuing, job Kj1 starts only after Ijn is done. Also, Until Ijn is done,all the worker nodes are only processing jobs related to input I thatwere added to the job queue ahead of the segments of video K.

In some embodiments, the master node may associate, with every videoreceived, a target speed of processing. For example, a processing speedof 5× implies that the job is to be done 5 times faster than real time.Next the jobs are segmented at key frames while maintaining the speedrequirements.

The Job Dispatcher may start by submitting a small number of segments,S, from every input into the job queue. The number S may be one or ahigher number based on the granularity of service layer agreement thatis to be met.

The worker nodes receive their respective encoding segments, and at theend of encoding a chunk, return the results of encoding back to themaster node.

The job dispatcher may keep a progress check for each job. The jobdispatcher may also monitor the processing time taken by each workernode for encoding the previously sent segments.

Based on the progress of each input, the desired speed of encoding forthat job and the available worker nodes, the job dispatcher dynamicallydecides the number of new segments to be submitted to the job queue.NumNewSegmentsToSubmit (I, S)=Status (Progress of input I, Number ofworker nodes)

Using this relation, for example, when the master node notices that agiven video is falling behind on the target speed of encoding, themaster node may submit more segments from that video to the job queue.Similarly, for another video, whose encoding is running faster than itstarget rate, fewer number of segments may be submitted to the queue.

FIG. 36 illustrates an example flowchart for a method 3600 of encodingmultiple videos in a distributed computer network.

The method 3600 includes, at 3602, receiving at least a portion of avideo. For example, the submission of segments described above may beginwithout having to wait until the entire video is received, e.g., simplyafter first S number of video segments are available for encoding.

The method 3600 includes, at 3604, splitting the portion into multiplevideo segments. In some embodiments, the received portion may be splitinto same sized video segments (e.g., 6 seconds long). In someembodiments, the techniques descried herein, e.g., for including keyframes in an open GOP scenario, may be used.

The method 3600 includes, at 3606, submitting the multiple videosegments to a video encoding pipeline that includes segments from atleast one other video, wherein the video encoding pipeline operates toissue video segments to encoding nodes for video encoding in a priorityorder. The priority order may depend on the monitoring of the completionspeed of encoding of the video.

The method 3600 includes, at 3608, adjusting a number of video segmentssent to the video encoding pipeline based on encoding performance.

In some embodiments, the adjustment is performed in response to a targetencoding time budget for the video and an encoding latency in thedistributed network of servers. For example, when a video is fallingbehind on a scheduled time budget, a greater number of segments of thatvideo may be submitted to the encoding pipeline.

In some embodiments, the encoding of the video is started after encodingof the at least one other video and the encoding of the video isfinished before the encoding of the at least one other video.

In some embodiments, the splitting operation includes splitting theportion into multiple video segments so that each video segment includesat least one key frame.

In some embodiments, at least some video segments include at least oneframe occurring temporally prior to the at least one key frame. In otherwords, the first frame in some segments may not be an independentlydecodable frame such as an I frame but may be a P or B type frame.

In some embodiments, the sending to the encoding node occurs prior toreceiving the entirety of the video at the server.

In some embodiments, the multiple video segments of the input video andsegments from the at least one other video may be of same size, and maythus include the number of video frames.

In some embodiments, the method 3600 further includes selecting, basedon a video encoding load, the one of the encoding nodes to which the atleast one of the multiple video segments is sent out of order.

It will be appreciated that techniques are disclosed for prioritizingsubmissions of video encoding segments to a pipeline of video encodingsuch that encoding jobs from one video being encoded do not blockencoding of another video segment. By using the target completion rateor time budget as criteria, the queue filling technique ensures thatpriority of encoding is increased when a video encoding may fall behindits schedule.

Computer System

FIG. 37 is a block diagram of a computer system as may be used toimplement features of some of the embodiments, e.g., master nodes orworker nodes, as described herein. The computing system 3700 may includeone or more central processing units (“processors”) 3705, memory 3710,input/output devices 3725 (e.g., keyboard and pointing devices, displaydevices), storage devices 3720 (e.g., disk drives), and network adapters3730 (e.g., network interfaces) that are connected to an interconnect3715. The interconnect 3715 is illustrated as an abstraction thatrepresents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The interconnect 3715, therefore, may include, for example,a system bus, a Peripheral Component Interconnect (PCI) bus orPCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The memory 3710 and storage devices 3720 are computer-readable storagemedia that may store instructions that implement at least portions ofthe various embodiments. In addition, the data structures and messagestructures may be stored or transmitted via a data transmission medium,e.g., a signal on a communications link. Various communications linksmay be used, e.g., the Internet, a local area network, a wide areanetwork, or a point-to-point dial-up connection. Thus, computer readablemedia can include computer-readable storage media (e.g., “nontransitory” media) and computer-readable transmission media.

The instructions stored in memory 3710 can be implemented as softwareand/or firmware to program the processor(s) 3705 to carry out actionsdescribed above. In some embodiments, such software or firmware may beinitially provided to the processing system 3700 by downloading it froma remote system through the computing system 3700 (e.g., via networkadapter 3730).

It will be appreciated by one of ordinary skill in the art that thedistributed video encoding techniques described in the present documentcan be used to accelerate video encoding by breaking an input video upinto multiple segments and then assigning segments to differentcomputers to perform the encoding task. In one advantageous aspect, thesystem can not only perform effective variable bits-per-frame encodingwithin a single segment, but cannot achieve good encoding acrossmultiple segments as each computer is only working on a single segment.One advantageous aspect is explained with reference to FIG. 38.

In FIG. 38, graph 3802 shows the complexity of a typical input videowith a high complexity region in the middle of the video as a functionof time along the horizontal axis (3804) and encoding complexity of thevideo along the vertical axis 3806.

Graph 3820 shows the video bitrate allocation that will occur in a goodVBR system. The high complexity region of the video is allocatedadditional bits in order to deliver good video quality during this time,with the average bitrate allocation across all frames of the videodepicted by dashed line 3822.

Graph 3840 shows the same video complexity as in 3806, in a distributedvideo encoding system and shows where the video will be segmented alongtime axis 3804, in this case, into 3 segments, represented by verticaldashed lines.

Graph 3860 shows the video bitrate allocation that will occur in a priorart distributed video encoding system. Each segment encoder will achievethe average bitrate for the segment, and will allocate bits to the highand low complexity sections of the video. As can be seen, this bitrateallocation is not globally optimized compared to the graph 3820. Theabove-discussed techniques enable a distributed video encoding system toachieve a performance similar to 3820, by performing complexity analysisof segments, and by using the results of segment-wise complexityanalysis during encoding of each segment.

The various embodiments introduced herein can be implemented by, forexample, programmable circuitry (e.g., one or more microprocessors)programmed with software and/or firmware, or entirely in special-purposehardwired (non-programmable) circuitry, or in a combination of suchforms. Special-purpose hardwired circuitry may be in the form of, forexample, one or more ASICs, PLDs, FPGAs, etc.

Remarks

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding of the disclosure. However, in certaininstances, well-known details are not described in order to avoidobscuring the description. Further, various modifications may be madewithout deviating from the scope of the embodiments. Accordingly, theembodiments are not limited except as by the appended claims.

Reference in this specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the disclosure. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment, nor are separate or alternative embodimentsmutually exclusive of other embodiments. Moreover, various features aredescribed which may be exhibited by some embodiments and not by others.Similarly, various requirements are described which may be requirementsfor some embodiments but not for other embodiments.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the disclosure, and in thespecific context where each term is used. Certain terms that are used todescribe the disclosure are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the disclosure. For convenience, certainterms may be highlighted, for example using italics and/or quotationmarks. The use of highlighting has no influence on the scope and meaningof a term; the scope and meaning of a term is the same, in the samecontext, whether or not it is highlighted. It will be appreciated thatthe same thing can be said in more than one way. One will recognize that“memory” is one form of a “storage” and that the terms may on occasionbe used interchangeably.

Consequently, alternative language and synonyms may be used for any oneor more of the terms discussed herein, nor is any special significanceto be placed upon whether or not a term is elaborated or discussedherein. Synonyms for certain terms are provided. A recital of one ormore synonyms does not exclude the use of other synonyms. The use ofexamples anywhere in this specification including examples of any termdiscussed herein is illustrative only, and is not intended to furtherlimit the scope and meaning of the disclosure or of any exemplifiedterm. Likewise, the disclosure is not limited to various embodimentsgiven in this specification.

Without intent to further limit the scope of the disclosure, examples ofinstruments, apparatus, methods and their related results according tothe embodiments of the present disclosure are given above. Note thattitles or subtitles may be used in the examples for convenience of areader, which in no way should limit the scope of the disclosure. Unlessotherwise defined, all technical and scientific terms used herein havethe same meaning as commonly understood by one of ordinary skill in theart to which this disclosure pertains. In the case of conflict, thepresent document, including definitions will control.

What is claimed is:
 1. A computer-implemented method, implemented at aserver in a distributed network of servers, comprising: receiving atleast a portion of a video; associating a target encoding speed with thevideo; splitting the portion into multiple overlapping video segments;submitting a first number of segments from the multiple overlappingvideo segments to a video encoding pipeline that includes segments fromat least one other video, wherein the submitting additionally provides anumber of frames to skip from encoding for each of the first number ofsegments, wherein the video encoding pipeline operates to issue videosegments to encoding nodes for video encoding in a priority order;comparing an encoding speed of the first number of segments with thetarget encoding speed; and based on the comparing, dynamicallydetermining a second number of segments from the multiple overlappingvideo segments to submit to the video encoding pipeline such that, whenthe encoding speed is less than the target encoding speed, the secondnumber of segments is determined to be more than the first number ofsegments, and when the encoding speed is greater than the targetencoding speed, then the second number of segments is determined to beless than the first number of segments.
 2. The computer-implementedmethod of claim 1, wherein the dynamically determining is performed inresponse to a target encoding time budget for the video and an encodinglatency in the distributed network of servers.
 3. Thecomputer-implemented method of claim 1, wherein encoding of the video isstarted after encoding of the at least one other video and the encodingof the video is finished before the encoding of the at least one othervideo.
 4. The computer-implemented method of claim 1, wherein thesplitting operation includes splitting the portion into the multipleoverlapping video segments such that each video segment includes atleast one key frame.
 5. The computer-implemented method of claim 4,wherein at least some of the multiple overlapping video segments includeat least one frame occurring temporally prior to the at least one keyframe.
 6. The computer-implemented method of claim 1 further comprising:issuing to the encoding nodes occurs prior to receiving an entirety ofthe video at the server.
 7. The computer-implemented method of claim 1,wherein the multiple overlapping video segments of the video and thesegments from the at least one other video include a same number ofvideo frames.
 8. The computer-implemented method of claim 1, furthercomprising: selecting, based on a video encoding load, the one of theencoding nodes to which the at least one of the multiple overlappingvideo segments is sent using adjusted queue submission.
 9. Anon-transitory computer-readable medium storing instructions that, uponexecution, cause at least one computer processor to perform a methodcomprising: receiving at least a portion of a video; associating atarget encoding speed with the video; splitting the portion intomultiple overlapping video segments; submitting a first number ofsegments from the multiple overlapping video segments to a videoencoding pipeline that includes segments from at least one other video,wherein the submitting additionally provides a number of frames to skipfrom encoding for each of the first number of segments, wherein thevideo encoding pipeline operates to issue video segments to encodingnodes for video encoding in a priority order; comparing an encodingspeed of the first number of segments with the target encoding speed;and based on the comparing, dynamically determining a second number ofsegments from the multiple overlapping video segments to submit to thevideo encoding pipeline such that when the encoding speed is less thanthe target encoding speed, then the second number of segments isdetermined to be more than the first number of segments, and when theencoding speed is greater than the target encoding speed, then thesecond number of segments is determined to be less that the first numberof segments.
 10. The non-transitory computer-readable medium of claim 9,further comprising: changing the priority order in response to a targetencoding time budget for the video and an encoding latency in adistributed network of servers.
 11. The non-transitory computer-readablemedium of claim 9, wherein encoding of the video is started afterencoding of the at least one other video and the encoding of the videois finished before the encoding of the at least one other video.
 12. Thenon-transitory computer-readable medium of claim 9, wherein thesplitting includes splitting the portion into the multiple overlappingvideo segments such that each video segment includes at least one keyframe.
 13. The non-transitory computer-readable medium of claim 12,wherein at least some of the multiple overlapping video segments includeat least one frame occurring temporally prior to the at least one keyframe.
 14. The non-transitory computer-readable medium of claim 9,further comprising: issuing to the encoding node include instructionsfor sending prior to receiving an entirety of the video at a server. 15.A computer system, comprising: at least one processor; at least onememory comprising instructions configured to cause the at least oneprocessor to perform a method, comprising: receiving at least a portionof a video; associating a target encoding speed with the video;splitting the portion into multiple overlapping video segments;submitting a first number of segments from the multiple overlappingvideo segments to a video encoding pipeline that includes segments fromat least one other video, wherein the submitting additionally provides anumber of frames to skip from encoding for each of the first number ofsegments, wherein the video encoding pipeline operates to issue videosegments to encoding nodes for video encoding in a priority order;comparing an encoding speed of the first number of segments with thetarget encoding speed; and based on the comparing, dynamicallydetermining a second number of segments from the multiple overlappingvideo segments to submit to the video encoding pipeline such that whenthe encoding speed is less than the target encoding speed, then thesecond number of segments is determined to be more than the first numberof segments, and when the encoding speed is greater than the targetencoding speed, then the second number of segments is determined to beless that the first number of segments.
 16. The computer system of claim15, wherein the priority order is changed in response to a targetencoding time budget for the video and an encoding latency in adistributed network of servers.
 17. The computer system of claim 15,wherein encoding of the video is started after encoding of the at leastone other video and the encoding of the video is finished before theencoding of the at least one other video.
 18. The computer system ofclaim 17, wherein the splitting includes splitting the portion into themultiple overlapping video segments such that each video segmentincludes at least one key frame.
 19. The computer system of claim 18,wherein at least some of the multiple overlapping video segments includeat least one frame occurring temporally prior to the at least one keyframe.
 20. The computer system of claim 19, wherein the video segmentsare submitted to the encoding node prior to receiving an entirety of thevideo at a server.